Nov 02 22:47:21.318431 ip-10-0-135-176 systemd[1]: Starting Kubernetes Kubelet... Nov 02 22:47:21.699098 ip-10-0-135-176 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:47:21.699098 ip-10-0-135-176 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 22:47:21.699098 ip-10-0-135-176 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:47:21.699098 ip-10-0-135-176 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:47:21.705256 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705172 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:47:21.705256 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705252 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:47:21.705256 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705256 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:47:21.705256 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705259 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:47:21.705256 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705262 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:47:21.705433 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705264 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:47:21.705433 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705267 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:47:21.705433 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705270 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:47:21.705433 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705273 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:47:21.705433 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705275 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:47:21.705433 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705278 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:47:21.705433 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705281 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:47:21.705433 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705289 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:47:21.705433 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705292 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:47:21.705433 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705295 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:47:21.705433 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705297 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:47:21.705433 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705300 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:47:21.705433 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705302 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:47:21.705433 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705305 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:47:21.705433 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705308 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:47:21.705433 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705310 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:47:21.705433 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705313 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:47:21.705433 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705317 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:47:21.705433 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705321 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:47:21.705884 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705325 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:47:21.705884 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705327 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:47:21.705884 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705330 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:47:21.705884 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705332 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:47:21.705884 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705335 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:47:21.705884 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705338 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:47:21.705884 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705342 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:47:21.705884 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705345 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:47:21.705884 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705348 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:47:21.705884 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705351 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:47:21.705884 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705354 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:47:21.705884 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705357 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:47:21.705884 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705360 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:47:21.705884 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705363 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:47:21.705884 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705368 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:47:21.705884 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705370 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:47:21.705884 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705373 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:47:21.705884 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705376 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:47:21.705884 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705378 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:47:21.706353 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705381 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:47:21.706353 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705389 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:47:21.706353 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705391 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:47:21.706353 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705394 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:47:21.706353 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705396 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:47:21.706353 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705399 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:47:21.706353 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705401 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:47:21.706353 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705404 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:47:21.706353 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705407 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:47:21.706353 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705409 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:47:21.706353 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705411 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:47:21.706353 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705414 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:47:21.706353 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705416 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:47:21.706353 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705419 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:47:21.706353 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705421 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:47:21.706353 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705424 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:47:21.706353 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705427 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:47:21.706353 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705429 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:47:21.706353 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705432 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:47:21.706353 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705434 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:47:21.706854 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705437 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:47:21.706854 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705439 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:47:21.706854 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705443 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:47:21.706854 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705445 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:47:21.706854 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705448 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:47:21.706854 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705450 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:47:21.706854 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705453 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:47:21.706854 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705456 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:47:21.706854 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705458 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:47:21.706854 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705461 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:47:21.706854 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705464 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:47:21.706854 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705466 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:47:21.706854 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705469 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:47:21.706854 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705471 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:47:21.706854 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705474 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:47:21.706854 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705476 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:47:21.706854 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705478 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:47:21.706854 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705481 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:47:21.706854 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705483 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:47:21.706854 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705486 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:47:21.707341 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705488 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:47:21.707341 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705491 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:47:21.707341 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705493 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:47:21.707341 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705496 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:47:21.707341 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705499 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:47:21.707341 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705501 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:47:21.707341 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705504 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:47:21.707341 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705506 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:47:21.707341 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705509 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:47:21.707341 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705511 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:47:21.707341 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705514 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:47:21.707341 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705516 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:47:21.707341 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705518 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:47:21.707341 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705521 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:47:21.707341 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705523 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:47:21.707341 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705526 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:47:21.707341 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705529 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:47:21.707341 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705531 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:47:21.707341 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705535 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:47:21.707785 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705538 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:47:21.707785 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705541 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:47:21.707785 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705543 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:47:21.707785 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.705546 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:47:21.707785 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706423 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:47:21.707785 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706430 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:47:21.707785 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706433 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:47:21.707785 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706436 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:47:21.707785 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706439 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:47:21.707785 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706441 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:47:21.707785 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706444 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:47:21.707785 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706446 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:47:21.707785 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706449 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:47:21.707785 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706451 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:47:21.707785 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706454 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:47:21.707785 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706456 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:47:21.707785 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706459 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:47:21.707785 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706461 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:47:21.707785 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706464 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:47:21.707785 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706467 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:47:21.708277 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706469 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:47:21.708277 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706472 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:47:21.708277 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706474 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:47:21.708277 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706477 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:47:21.708277 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706479 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:47:21.708277 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706482 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:47:21.708277 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706485 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:47:21.708277 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706487 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:47:21.708277 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706490 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:47:21.708277 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706504 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:47:21.708277 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706509 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:47:21.708277 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706513 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:47:21.708277 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706517 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:47:21.708277 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706519 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:47:21.708277 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706522 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:47:21.708277 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706524 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:47:21.708277 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706529 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:47:21.708277 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706533 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:47:21.708277 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706536 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:47:21.708728 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706540 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:47:21.708728 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706542 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:47:21.708728 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706545 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:47:21.708728 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706548 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:47:21.708728 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706551 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:47:21.708728 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706554 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:47:21.708728 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706556 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:47:21.708728 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706558 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:47:21.708728 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706561 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:47:21.708728 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706564 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:47:21.708728 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706566 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:47:21.708728 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706569 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:47:21.708728 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706571 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:47:21.708728 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706574 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:47:21.708728 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706576 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:47:21.708728 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706579 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:47:21.708728 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706581 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:47:21.708728 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706584 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:47:21.708728 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706587 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:47:21.708728 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706590 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:47:21.709244 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706592 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:47:21.709244 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706595 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:47:21.709244 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706597 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:47:21.709244 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706606 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:47:21.709244 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706609 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:47:21.709244 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706612 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:47:21.709244 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706614 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:47:21.709244 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706616 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:47:21.709244 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706619 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:47:21.709244 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706622 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:47:21.709244 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706624 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:47:21.709244 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706626 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:47:21.709244 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706629 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:47:21.709244 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706632 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:47:21.709244 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706634 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:47:21.709244 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706637 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:47:21.709244 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706639 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:47:21.709244 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706642 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:47:21.709244 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706644 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:47:21.709244 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706646 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:47:21.709717 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706649 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:47:21.709717 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706651 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:47:21.709717 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706654 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:47:21.709717 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706656 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:47:21.709717 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706659 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:47:21.709717 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706661 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:47:21.709717 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706664 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:47:21.709717 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706666 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:47:21.709717 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706668 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:47:21.709717 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706671 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:47:21.709717 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706674 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:47:21.709717 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706676 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:47:21.709717 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706679 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:47:21.709717 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706681 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:47:21.709717 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706684 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:47:21.709717 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706686 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:47:21.709717 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706695 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:47:21.709717 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706697 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:47:21.709717 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706700 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:47:21.710175 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706702 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:47:21.710175 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706705 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:47:21.710175 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706707 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:47:21.710175 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706710 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:47:21.710175 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706712 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:47:21.710175 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706714 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:47:21.710175 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706717 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:47:21.710175 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706720 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:47:21.710175 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706726 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:47:21.710175 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706729 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:47:21.710175 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706731 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:47:21.710175 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.706734 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:47:21.710175 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707313 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:47:21.710175 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707319 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:47:21.710175 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707322 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:47:21.710175 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707324 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:47:21.710175 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707327 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:47:21.710175 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707330 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:47:21.710175 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707332 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:47:21.710624 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707335 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:47:21.710624 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707337 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:47:21.710624 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707340 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:47:21.710624 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707343 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:47:21.710624 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707346 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:47:21.710624 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707349 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:47:21.710624 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707352 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:47:21.710624 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707354 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:47:21.710624 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707357 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:47:21.710624 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707359 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:47:21.710624 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707367 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:47:21.710624 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707370 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:47:21.710624 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707372 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:47:21.710624 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707375 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:47:21.710624 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707377 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:47:21.710624 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707380 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:47:21.710624 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707382 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:47:21.710624 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707385 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:47:21.710624 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707387 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:47:21.711085 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707390 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:47:21.711085 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707393 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:47:21.711085 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707395 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:47:21.711085 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707398 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:47:21.711085 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707400 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:47:21.711085 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707403 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:47:21.711085 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707405 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:47:21.711085 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707408 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:47:21.711085 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707410 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:47:21.711085 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707413 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:47:21.711085 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707415 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:47:21.711085 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707417 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:47:21.711085 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707420 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:47:21.711085 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707423 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:47:21.711085 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707425 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:47:21.711085 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707428 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:47:21.711085 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707431 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:47:21.711085 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707433 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:47:21.711085 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707435 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:47:21.711551 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707439 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:47:21.711551 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707443 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:47:21.711551 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707445 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:47:21.711551 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707447 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:47:21.711551 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707450 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:47:21.711551 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707458 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:47:21.711551 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707462 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:47:21.711551 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707467 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:47:21.711551 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707469 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:47:21.711551 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707472 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:47:21.711551 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707475 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:47:21.711551 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707478 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:47:21.711551 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707482 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:47:21.711551 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707485 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:47:21.711551 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707487 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:47:21.711551 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707490 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:47:21.711551 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707495 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:47:21.711551 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707498 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:47:21.711551 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707501 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:47:21.711993 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707504 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:47:21.711993 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707507 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:47:21.711993 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707509 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:47:21.711993 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707512 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:47:21.711993 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707515 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:47:21.711993 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707518 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:47:21.711993 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707521 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:47:21.711993 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707523 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:47:21.711993 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707526 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:47:21.711993 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707528 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:47:21.711993 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707531 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:47:21.711993 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707533 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:47:21.711993 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707536 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:47:21.711993 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707538 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:47:21.711993 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707541 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:47:21.711993 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707543 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:47:21.711993 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707546 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:47:21.711993 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707548 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:47:21.711993 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707550 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:47:21.711993 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707561 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:47:21.712470 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707564 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:47:21.712470 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707567 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:47:21.712470 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707569 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:47:21.712470 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707572 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:47:21.712470 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707574 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:47:21.712470 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707578 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:47:21.712470 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707581 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:47:21.712470 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707583 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:47:21.712470 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707586 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:47:21.712470 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707588 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:47:21.712470 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707591 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:47:21.712470 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707593 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:47:21.712470 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707596 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:47:21.712470 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707600 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:47:21.712470 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707602 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:47:21.712470 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707605 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:47:21.712470 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707607 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:47:21.712470 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707611 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:47:21.712470 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707613 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:47:21.712470 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707615 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:47:21.712928 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707618 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:47:21.712928 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.707620 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:47:21.712928 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709205 2573 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 22:47:21.712928 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709216 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 22:47:21.712928 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709228 2573 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 22:47:21.712928 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709232 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 22:47:21.712928 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709236 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 22:47:21.712928 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709240 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 22:47:21.712928 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709244 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 22:47:21.712928 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709249 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 22:47:21.712928 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709252 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 22:47:21.712928 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709255 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 22:47:21.712928 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709267 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 22:47:21.712928 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709271 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 22:47:21.712928 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709274 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 22:47:21.712928 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709277 2573 flags.go:64] FLAG: --cgroup-root="" Nov 02 22:47:21.712928 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709280 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 22:47:21.712928 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709284 2573 flags.go:64] FLAG: --client-ca-file="" Nov 02 22:47:21.712928 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709287 2573 flags.go:64] FLAG: --cloud-provider="external" Nov 02 22:47:21.712928 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709290 2573 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 22:47:21.712928 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709297 2573 flags.go:64] FLAG: --cluster-domain="" Nov 02 22:47:21.712928 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709300 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 22:47:21.712928 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709303 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 22:47:21.713499 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709306 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 22:47:21.713499 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709310 2573 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 22:47:21.713499 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709314 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 22:47:21.713499 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709317 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 22:47:21.713499 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709320 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 22:47:21.713499 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709323 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 22:47:21.713499 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709326 2573 flags.go:64] FLAG: --contention-profiling="false" Nov 02 22:47:21.713499 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709329 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 22:47:21.713499 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709332 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 22:47:21.713499 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709335 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 22:47:21.713499 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709338 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 22:47:21.713499 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709342 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 22:47:21.713499 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709345 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 22:47:21.713499 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709348 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 22:47:21.713499 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709350 2573 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 22:47:21.713499 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709353 2573 flags.go:64] FLAG: --enable-server="true" Nov 02 22:47:21.713499 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709356 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 22:47:21.713499 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709362 2573 flags.go:64] FLAG: --event-burst="100" Nov 02 22:47:21.713499 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709365 2573 flags.go:64] FLAG: --event-qps="50" Nov 02 22:47:21.713499 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709368 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 22:47:21.713499 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709371 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 22:47:21.713499 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709374 2573 flags.go:64] FLAG: --eviction-hard="" Nov 02 22:47:21.713499 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709384 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 22:47:21.713499 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709387 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 22:47:21.714192 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709390 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 22:47:21.714192 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709393 2573 flags.go:64] FLAG: --eviction-soft="" Nov 02 22:47:21.714192 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709396 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 22:47:21.714192 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709398 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 22:47:21.714192 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709401 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 22:47:21.714192 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709404 2573 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 22:47:21.714192 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709407 2573 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 22:47:21.714192 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709410 2573 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 22:47:21.714192 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709412 2573 flags.go:64] FLAG: --feature-gates="" Nov 02 22:47:21.714192 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709416 2573 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 22:47:21.714192 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709419 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 22:47:21.714192 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709422 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 22:47:21.714192 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709425 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 22:47:21.714192 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709428 2573 flags.go:64] FLAG: --healthz-port="10248" Nov 02 22:47:21.714192 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709431 2573 flags.go:64] FLAG: --help="false" Nov 02 22:47:21.714192 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709434 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:47:21.714192 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709437 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 22:47:21.714192 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709440 2573 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 22:47:21.714192 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709444 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 22:47:21.714192 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709447 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 22:47:21.714192 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709450 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 22:47:21.714192 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709453 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 22:47:21.714192 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709456 2573 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 22:47:21.714745 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709459 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 22:47:21.714745 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709461 2573 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 22:47:21.714745 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709464 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 22:47:21.714745 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709467 2573 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 22:47:21.714745 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709470 2573 flags.go:64] FLAG: --kube-reserved="" Nov 02 22:47:21.714745 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709473 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 22:47:21.714745 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709476 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 22:47:21.714745 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709478 2573 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 22:47:21.714745 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709487 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 22:47:21.714745 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709490 2573 flags.go:64] FLAG: --lock-file="" Nov 02 22:47:21.714745 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709493 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 22:47:21.714745 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709496 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 22:47:21.714745 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709499 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 22:47:21.714745 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709504 2573 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 22:47:21.714745 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709507 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 22:47:21.714745 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709510 2573 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 22:47:21.714745 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709513 2573 flags.go:64] FLAG: --logging-format="text" Nov 02 22:47:21.714745 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709515 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 22:47:21.714745 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709519 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 22:47:21.714745 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709521 2573 flags.go:64] FLAG: --manifest-url="" Nov 02 22:47:21.714745 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709524 2573 flags.go:64] FLAG: --manifest-url-header="" Nov 02 22:47:21.714745 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709528 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 22:47:21.714745 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709531 2573 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 22:47:21.714745 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709535 2573 flags.go:64] FLAG: --max-pods="110" Nov 02 22:47:21.714745 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709538 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 22:47:21.715348 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709541 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 22:47:21.715348 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709544 2573 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 22:47:21.715348 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709546 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 22:47:21.715348 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709549 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 22:47:21.715348 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709552 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 22:47:21.715348 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709555 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 22:47:21.715348 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709561 2573 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 22:47:21.715348 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709564 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 22:47:21.715348 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709567 2573 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 22:47:21.715348 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709570 2573 flags.go:64] FLAG: --pod-cidr="" Nov 02 22:47:21.715348 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709573 2573 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 22:47:21.715348 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709576 2573 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 22:47:21.715348 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709579 2573 flags.go:64] FLAG: --pods-per-core="0" Nov 02 22:47:21.715348 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709582 2573 flags.go:64] FLAG: --port="10250" Nov 02 22:47:21.715348 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709584 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 22:47:21.715348 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709587 2573 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0fed1cfa20bbd3c95" Nov 02 22:47:21.715348 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709596 2573 flags.go:64] FLAG: --qos-reserved="" Nov 02 22:47:21.715348 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709599 2573 flags.go:64] FLAG: --read-only-port="10255" Nov 02 22:47:21.715348 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709602 2573 flags.go:64] FLAG: --register-node="true" Nov 02 22:47:21.715348 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709605 2573 flags.go:64] FLAG: --register-with-taints="" Nov 02 22:47:21.715348 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709609 2573 flags.go:64] FLAG: --registry-burst="10" Nov 02 22:47:21.715348 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709611 2573 flags.go:64] FLAG: --registry-qps="5" Nov 02 22:47:21.715348 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709614 2573 flags.go:64] FLAG: --reserved-cpus="" Nov 02 22:47:21.715348 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709617 2573 flags.go:64] FLAG: --reserved-memory="" Nov 02 22:47:21.715348 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709621 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 22:47:21.715348 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709624 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 22:47:21.715984 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709627 2573 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 22:47:21.715984 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709629 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 22:47:21.715984 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709632 2573 flags.go:64] FLAG: --runonce="false" Nov 02 22:47:21.715984 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709635 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 22:47:21.715984 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709638 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 22:47:21.715984 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709640 2573 flags.go:64] FLAG: --seccomp-default="false" Nov 02 22:47:21.715984 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709649 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 22:47:21.715984 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709652 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 22:47:21.715984 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709655 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 22:47:21.715984 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709658 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 22:47:21.715984 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709661 2573 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 22:47:21.715984 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709664 2573 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 22:47:21.715984 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709666 2573 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 22:47:21.715984 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709669 2573 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 22:47:21.715984 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709672 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 22:47:21.715984 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709675 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 22:47:21.715984 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709677 2573 flags.go:64] FLAG: --system-cgroups="" Nov 02 22:47:21.715984 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709682 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 22:47:21.715984 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709687 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 22:47:21.715984 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709690 2573 flags.go:64] FLAG: --tls-cert-file="" Nov 02 22:47:21.715984 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709693 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 22:47:21.715984 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709702 2573 flags.go:64] FLAG: --tls-min-version="" Nov 02 22:47:21.715984 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709705 2573 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 22:47:21.715984 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709716 2573 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 22:47:21.715984 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709719 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 22:47:21.716959 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709722 2573 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 22:47:21.716959 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709725 2573 flags.go:64] FLAG: --v="2" Nov 02 22:47:21.716959 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709729 2573 flags.go:64] FLAG: --version="false" Nov 02 22:47:21.716959 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709735 2573 flags.go:64] FLAG: --vmodule="" Nov 02 22:47:21.716959 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709739 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 22:47:21.716959 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.709742 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 22:47:21.716959 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.709943 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:47:21.716959 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.709947 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:47:21.716959 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.709951 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:47:21.716959 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.709953 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:47:21.716959 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.709956 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:47:21.716959 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.709958 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:47:21.716959 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.709961 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:47:21.716959 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.709964 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:47:21.716959 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.709966 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:47:21.716959 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.709968 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:47:21.716959 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.709971 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:47:21.716959 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.709973 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:47:21.716959 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.709976 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:47:21.716959 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.709978 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:47:21.716959 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.709981 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:47:21.717621 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.709983 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:47:21.717621 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.709986 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:47:21.717621 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.709988 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:47:21.717621 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.709992 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:47:21.717621 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.709994 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:47:21.717621 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.709997 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:47:21.717621 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.709999 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:47:21.717621 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710002 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:47:21.717621 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710004 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:47:21.717621 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710006 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:47:21.717621 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710013 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:47:21.717621 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710016 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:47:21.717621 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710019 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:47:21.717621 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710021 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:47:21.717621 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710025 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:47:21.717621 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710027 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:47:21.717621 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710030 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:47:21.717621 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710032 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:47:21.717621 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710035 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:47:21.717621 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710038 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:47:21.718103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710040 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:47:21.718103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710042 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:47:21.718103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710045 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:47:21.718103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710048 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:47:21.718103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710050 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:47:21.718103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710052 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:47:21.718103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710056 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:47:21.718103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710060 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:47:21.718103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710063 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:47:21.718103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710066 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:47:21.718103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710068 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:47:21.718103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710071 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:47:21.718103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710073 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:47:21.718103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710076 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:47:21.718103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710078 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:47:21.718103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710082 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:47:21.718103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710084 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:47:21.718103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710087 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:47:21.718103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710089 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:47:21.718103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710092 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:47:21.718683 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710094 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:47:21.718683 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710097 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:47:21.718683 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710099 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:47:21.718683 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710105 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:47:21.718683 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710108 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:47:21.718683 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710110 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:47:21.718683 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710114 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:47:21.718683 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710117 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:47:21.718683 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710120 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:47:21.718683 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710122 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:47:21.718683 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710138 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:47:21.718683 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710141 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:47:21.718683 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710143 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:47:21.718683 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710146 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:47:21.718683 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710148 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:47:21.718683 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710151 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:47:21.718683 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710153 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:47:21.718683 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710156 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:47:21.718683 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710159 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:47:21.719154 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710161 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:47:21.719154 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710164 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:47:21.719154 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710166 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:47:21.719154 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710169 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:47:21.719154 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710171 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:47:21.719154 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710174 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:47:21.719154 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710177 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:47:21.719154 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710179 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:47:21.719154 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710183 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:47:21.719154 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710185 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:47:21.719154 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710188 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:47:21.719154 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710191 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:47:21.719154 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710193 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:47:21.719154 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710195 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:47:21.719154 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710198 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:47:21.719154 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710200 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:47:21.719154 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710203 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:47:21.719154 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710211 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:47:21.719154 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710214 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:47:21.719154 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710218 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:47:21.719627 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710220 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:47:21.719627 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710223 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:47:21.719627 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710226 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:47:21.719627 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710228 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:47:21.719627 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710231 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:47:21.719627 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710233 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:47:21.719627 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710235 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:47:21.719627 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710238 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:47:21.719627 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710240 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:47:21.719627 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710243 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:47:21.719627 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710247 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:47:21.719627 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710251 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:47:21.719627 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.710279 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:47:21.719627 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710325 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:47:21.719627 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710330 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:47:21.719982 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710333 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:47:21.719982 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710335 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:47:21.719982 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710338 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:47:21.719982 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710340 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:47:21.719982 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710343 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:47:21.719982 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710347 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:47:21.719982 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710349 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:47:21.719982 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710352 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:47:21.719982 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710354 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:47:21.719982 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710356 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:47:21.719982 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710359 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:47:21.719982 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710362 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:47:21.719982 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710364 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:47:21.719982 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710366 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:47:21.719982 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710369 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:47:21.719982 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710372 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:47:21.719982 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710375 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:47:21.719982 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710379 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:47:21.719982 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710381 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:47:21.719982 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710384 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:47:21.720469 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710386 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:47:21.720469 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710389 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:47:21.720469 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710391 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:47:21.720469 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710394 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:47:21.720469 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710396 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:47:21.720469 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710398 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:47:21.720469 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710401 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:47:21.720469 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710403 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:47:21.720469 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710406 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:47:21.720469 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710408 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:47:21.720469 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710412 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:47:21.720469 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710415 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:47:21.720469 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710418 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:47:21.720469 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710422 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:47:21.720469 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710424 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:47:21.720469 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710427 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:47:21.720469 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710430 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:47:21.720469 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710434 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:47:21.720469 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710437 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:47:21.720987 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710440 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:47:21.720987 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710443 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:47:21.720987 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710445 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:47:21.720987 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710465 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:47:21.720987 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710469 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:47:21.720987 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710472 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:47:21.720987 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710475 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:47:21.720987 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710477 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:47:21.720987 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710480 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:47:21.720987 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710483 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:47:21.720987 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710487 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:47:21.720987 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710490 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:47:21.720987 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710492 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:47:21.720987 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710495 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:47:21.720987 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710497 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:47:21.720987 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710500 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:47:21.720987 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710502 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:47:21.720987 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710505 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:47:21.720987 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710507 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:47:21.720987 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710510 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:47:21.721473 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710513 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:47:21.721473 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710515 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:47:21.721473 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710517 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:47:21.721473 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710520 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:47:21.721473 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710522 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:47:21.721473 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710526 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:47:21.721473 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710529 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:47:21.721473 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710532 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:47:21.721473 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710534 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:47:21.721473 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710537 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:47:21.721473 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710541 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:47:21.721473 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710544 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:47:21.721473 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710546 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:47:21.721473 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710548 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:47:21.721473 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710552 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:47:21.721473 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710555 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:47:21.721473 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710558 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:47:21.721473 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710562 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:47:21.721473 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710565 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:47:21.721473 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710567 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:47:21.721946 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710570 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:47:21.721946 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710572 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:47:21.721946 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710577 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:47:21.721946 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710580 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:47:21.721946 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710582 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:47:21.721946 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710585 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:47:21.721946 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710588 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:47:21.721946 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710590 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:47:21.721946 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710593 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:47:21.721946 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710595 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:47:21.721946 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710598 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:47:21.721946 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710600 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:47:21.721946 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710603 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:47:21.721946 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710605 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:47:21.721946 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710608 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:47:21.721946 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710610 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:47:21.721946 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710613 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:47:21.721946 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710615 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:47:21.721946 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710619 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:47:21.722400 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710621 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:47:21.722400 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710624 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:47:21.722400 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710627 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:47:21.722400 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710631 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:47:21.722400 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710633 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:47:21.722400 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.710636 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:47:21.722400 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.719212 2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 22:47:21.722400 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.719229 2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 22:47:21.722400 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.719253 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 22:47:21.722400 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.719259 2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 22:47:21.722400 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719331 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:47:21.722400 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719336 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:47:21.722400 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719340 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:47:21.722400 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719342 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:47:21.722400 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719345 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:47:21.722400 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719349 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:47:21.722400 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719351 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:47:21.722400 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719354 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:47:21.722400 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719356 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:47:21.722400 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719359 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:47:21.722869 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719362 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:47:21.722869 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719364 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:47:21.722869 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719367 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:47:21.722869 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719370 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:47:21.722869 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719372 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:47:21.722869 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719375 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:47:21.722869 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719377 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:47:21.722869 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719380 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:47:21.722869 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719382 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:47:21.722869 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719385 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:47:21.722869 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719387 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:47:21.722869 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719389 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:47:21.722869 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719392 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:47:21.722869 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719395 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:47:21.722869 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719397 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:47:21.722869 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719400 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:47:21.722869 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719406 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:47:21.722869 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719409 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:47:21.722869 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719411 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:47:21.722869 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719413 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:47:21.723368 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719416 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:47:21.723368 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719419 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:47:21.723368 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719423 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:47:21.723368 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719426 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:47:21.723368 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719430 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:47:21.723368 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719433 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:47:21.723368 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719435 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:47:21.723368 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719438 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:47:21.723368 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719441 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:47:21.723368 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719443 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:47:21.723368 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719446 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:47:21.723368 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719450 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:47:21.723368 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719454 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:47:21.723368 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719458 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:47:21.723368 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719461 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:47:21.723368 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719464 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:47:21.723368 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719466 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:47:21.723368 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719469 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:47:21.723368 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719472 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:47:21.723826 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719475 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:47:21.723826 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719477 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:47:21.723826 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719480 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:47:21.723826 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719482 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:47:21.723826 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719485 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:47:21.723826 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719487 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:47:21.723826 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719490 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:47:21.723826 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719492 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:47:21.723826 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719495 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:47:21.723826 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719497 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:47:21.723826 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719500 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:47:21.723826 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719503 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:47:21.723826 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719505 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:47:21.723826 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719508 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:47:21.723826 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719510 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:47:21.723826 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719513 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:47:21.723826 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719515 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:47:21.723826 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719518 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:47:21.723826 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719520 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:47:21.723826 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719523 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:47:21.724375 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719525 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:47:21.724375 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719528 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:47:21.724375 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719530 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:47:21.724375 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719533 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:47:21.724375 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719536 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:47:21.724375 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719538 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:47:21.724375 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719541 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:47:21.724375 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719543 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:47:21.724375 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719545 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:47:21.724375 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719548 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:47:21.724375 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719551 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:47:21.724375 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719553 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:47:21.724375 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719556 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:47:21.724375 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719558 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:47:21.724375 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719561 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:47:21.724375 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719563 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:47:21.724375 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719565 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:47:21.724375 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719568 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:47:21.724375 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719570 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:47:21.724811 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719573 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:47:21.724811 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719575 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:47:21.724811 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719577 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:47:21.724811 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719580 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:47:21.724811 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719582 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:47:21.724811 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719585 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:47:21.724811 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719588 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:47:21.724811 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719590 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:47:21.724811 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719593 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:47:21.724811 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719595 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:47:21.724811 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719597 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:47:21.724811 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719600 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:47:21.724811 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719602 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:47:21.724811 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719605 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:47:21.724811 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719607 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:47:21.724811 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719610 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:47:21.724811 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719612 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:47:21.724811 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719615 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:47:21.725238 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.719643 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:47:21.725238 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719801 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:47:21.725238 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719805 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:47:21.725238 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719808 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:47:21.725238 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719811 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:47:21.725238 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719814 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:47:21.725238 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719818 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:47:21.725238 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719821 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:47:21.725238 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719824 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:47:21.725238 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719827 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:47:21.725238 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719829 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:47:21.725238 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719832 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:47:21.725238 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719834 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:47:21.725238 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719837 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:47:21.725238 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719839 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:47:21.725238 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719842 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:47:21.725615 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719844 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:47:21.725615 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719847 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:47:21.725615 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719849 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:47:21.725615 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719852 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:47:21.725615 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719854 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:47:21.725615 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719857 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:47:21.725615 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719859 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:47:21.725615 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719862 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:47:21.725615 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719864 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:47:21.725615 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719867 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:47:21.725615 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719869 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:47:21.725615 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719872 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:47:21.725615 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719874 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:47:21.725615 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719876 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:47:21.725615 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719879 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:47:21.725615 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719882 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:47:21.725615 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719884 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:47:21.725615 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719887 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:47:21.725615 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719889 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:47:21.726097 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719892 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:47:21.726097 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719895 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:47:21.726097 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719897 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:47:21.726097 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719900 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:47:21.726097 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719903 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:47:21.726097 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719905 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:47:21.726097 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719907 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:47:21.726097 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719910 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:47:21.726097 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719912 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:47:21.726097 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719915 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:47:21.726097 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719917 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:47:21.726097 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719920 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:47:21.726097 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719922 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:47:21.726097 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719925 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:47:21.726097 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719928 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:47:21.726097 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719930 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:47:21.726097 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719933 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:47:21.726097 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719935 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:47:21.726097 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719938 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:47:21.726097 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719940 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:47:21.726650 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719943 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:47:21.726650 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719945 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:47:21.726650 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719948 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:47:21.726650 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719950 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:47:21.726650 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719954 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:47:21.726650 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719957 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:47:21.726650 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719960 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:47:21.726650 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719963 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:47:21.726650 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719966 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:47:21.726650 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719969 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:47:21.726650 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719971 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:47:21.726650 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719973 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:47:21.726650 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719976 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:47:21.726650 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719978 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:47:21.726650 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719980 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:47:21.726650 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719983 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:47:21.726650 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719985 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:47:21.726650 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719989 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:47:21.726650 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719991 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:47:21.727103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719994 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:47:21.727103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.719998 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:47:21.727103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720001 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:47:21.727103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720004 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:47:21.727103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720006 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:47:21.727103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720008 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:47:21.727103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720011 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:47:21.727103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720013 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:47:21.727103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720016 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:47:21.727103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720018 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:47:21.727103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720021 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:47:21.727103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720023 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:47:21.727103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720026 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:47:21.727103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720029 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:47:21.727103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720032 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:47:21.727103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720034 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:47:21.727103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720037 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:47:21.727103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720039 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:47:21.727103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720042 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:47:21.727103 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720044 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:47:21.727593 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720046 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:47:21.727593 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720049 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:47:21.727593 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720051 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:47:21.727593 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720054 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:47:21.727593 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720058 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:47:21.727593 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720061 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:47:21.727593 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720064 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:47:21.727593 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720067 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:47:21.727593 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720069 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:47:21.727593 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720072 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:47:21.727593 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720075 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:47:21.727593 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720078 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:47:21.727593 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720080 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:47:21.727593 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.720105 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:47:21.727593 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720152 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:47:21.727940 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720157 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:47:21.727940 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720160 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:47:21.727940 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720162 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:47:21.727940 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720166 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:47:21.727940 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720170 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:47:21.727940 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720172 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:47:21.727940 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720175 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:47:21.727940 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720178 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:47:21.727940 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720181 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:47:21.727940 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720183 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:47:21.727940 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720187 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:47:21.727940 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720189 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:47:21.727940 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720192 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:47:21.727940 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720195 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:47:21.727940 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720197 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:47:21.727940 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720200 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:47:21.727940 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720202 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:47:21.727940 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720205 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:47:21.727940 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720207 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:47:21.728486 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720210 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:47:21.728486 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720212 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:47:21.728486 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720215 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:47:21.728486 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720217 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:47:21.728486 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720219 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:47:21.728486 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720222 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:47:21.728486 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720224 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:47:21.728486 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720227 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:47:21.728486 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720229 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:47:21.728486 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720231 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:47:21.728486 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720234 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:47:21.728486 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720237 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:47:21.728486 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720242 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:47:21.728486 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720244 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:47:21.728486 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720247 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:47:21.728486 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720249 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:47:21.728486 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720252 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:47:21.728486 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720254 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:47:21.728486 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720257 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:47:21.728486 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720259 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:47:21.729110 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720262 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:47:21.729110 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720264 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:47:21.729110 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720266 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:47:21.729110 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720271 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:47:21.729110 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720275 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:47:21.729110 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720277 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:47:21.729110 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720281 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:47:21.729110 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720283 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:47:21.729110 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720286 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:47:21.729110 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720289 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:47:21.729110 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720291 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:47:21.729110 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720293 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:47:21.729110 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720296 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:47:21.729110 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720298 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:47:21.729110 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720301 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:47:21.729110 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720303 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:47:21.729110 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720306 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:47:21.729110 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720308 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:47:21.729110 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720310 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:47:21.729604 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720313 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:47:21.729604 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720316 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:47:21.729604 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720318 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:47:21.729604 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720321 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:47:21.729604 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720323 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:47:21.729604 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720327 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:47:21.729604 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720330 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:47:21.729604 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720332 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:47:21.729604 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720335 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:47:21.729604 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720337 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:47:21.729604 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720340 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:47:21.729604 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720342 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:47:21.729604 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720345 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:47:21.729604 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720347 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:47:21.729604 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720349 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:47:21.729604 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720352 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:47:21.729604 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720355 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:47:21.729604 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720358 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:47:21.729604 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720360 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:47:21.730087 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720363 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:47:21.730087 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720365 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:47:21.730087 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720368 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:47:21.730087 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720371 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:47:21.730087 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720373 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:47:21.730087 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720375 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:47:21.730087 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720378 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:47:21.730087 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720380 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:47:21.730087 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720383 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:47:21.730087 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720385 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:47:21.730087 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720388 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:47:21.730087 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720391 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:47:21.730087 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720393 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:47:21.730087 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720396 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:47:21.730087 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720398 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:47:21.730087 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720400 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:47:21.730087 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720403 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:47:21.730087 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720405 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:47:21.730087 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720410 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:47:21.730087 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720412 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:47:21.730646 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720415 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:47:21.730646 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720417 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:47:21.730646 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720420 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:47:21.730646 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720422 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:47:21.730646 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720426 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:47:21.730646 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720428 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:47:21.730646 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720430 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:47:21.730646 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:21.720433 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:47:21.730646 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.720519 2573 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 22:47:21.730646 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.722605 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 22:47:21.730646 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.724199 2573 server.go:1014] "Starting client certificate rotation" Nov 02 22:47:21.730646 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.724297 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 22:47:21.730646 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.724328 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 22:47:21.730646 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.725871 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 22:47:21.730646 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.728753 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 22:47:21.738601 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.738586 2573 log.go:25] "Validated CRI v1 runtime API" Nov 02 22:47:21.741786 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.741765 2573 log.go:25] "Validated CRI v1 image API" Nov 02 22:47:21.742740 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.742717 2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 22:47:21.746548 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.746523 2573 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 aa01b5c2-a734-4ab5-97a2-b17b8799dbc8:/dev/nvme0n1p3 e9619993-14af-4bf9-85f5-a712396b498f:/dev/nvme0n1p4] Nov 02 22:47:21.746606 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.746549 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 22:47:21.753526 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.753400 2573 manager.go:217] Machine: {Timestamp:2026-11-02 22:47:21.752487095 +0000 UTC m=+0.314850424 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099873 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f25bc58672d08509426113e404a38 SystemUUID:ec2f25bc-5867-2d08-5094-26113e404a38 BootID:f3231f38-0bcb-4949-9784-5daedbcd92ab Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:47:3c:9d:e1:f9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:47:3c:9d:e1:f9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:68:e3:b2:9c:15 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 22:47:21.754162 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.754151 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 22:47:21.754292 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.754279 2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 22:47:21.756973 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.756945 2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 22:47:21.757111 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.756975 2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-176.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 22:47:21.757184 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.757114 2573 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 22:47:21.757184 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.757122 2573 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 22:47:21.757184 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.757162 2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 22:47:21.757184 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.757176 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 22:47:21.757941 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.757931 2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 22:47:21.761535 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.761524 2573 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 22:47:21.761652 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.761643 2573 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 22:47:21.762396 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.762386 2573 kubelet.go:493] "Attempting to sync node with API server" Nov 02 22:47:21.762440 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.762400 2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 22:47:21.762440 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.762413 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 22:47:21.762440 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.762422 2573 kubelet.go:405] "Adding apiserver pod source" Nov 02 22:47:21.762440 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.762430 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 22:47:21.762766 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.762754 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 22:47:21.764755 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.764741 2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 22:47:21.769163 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.769150 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 22:47:21.769885 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.769873 2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 22:47:21.769919 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.769910 2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 22:47:21.770073 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.770063 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 22:47:21.770112 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.770079 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 22:47:21.770112 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.770086 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 22:47:21.770112 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.770092 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 22:47:21.770112 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.770097 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 22:47:21.770112 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.770103 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 22:47:21.770621 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.770613 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 22:47:21.770621 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.770622 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 22:47:21.770675 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.770629 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 22:47:21.770675 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.770641 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 22:47:21.770675 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.770654 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 22:47:21.770675 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.770663 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 22:47:21.771166 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.771158 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 22:47:21.771203 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.771169 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 22:47:21.774805 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.774784 2573 server.go:1263] "Started kubelet" Nov 02 22:47:21.774940 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.774901 2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 22:47:21.775058 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.774917 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 22:47:21.775120 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.775109 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 22:47:21.775645 ip-10-0-135-176 systemd[1]: Started Kubernetes Kubelet. Nov 02 22:47:21.776264 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.776185 2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 22:47:21.778061 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.778045 2573 server.go:317] "Adding debug handlers to kubelet server" Nov 02 22:47:21.780425 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:21.780408 2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 22:47:21.782762 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.782746 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 22:47:21.783516 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.783498 2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 22:47:21.783741 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:21.783717 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-176.us-west-1.compute.internal\" not found" Nov 02 22:47:21.784085 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.784067 2573 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 22:47:21.784194 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.784089 2573 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 22:47:21.784194 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.784069 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 22:47:21.784194 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.784182 2573 reconstruct.go:97] "Volume reconstruction finished" Nov 02 22:47:21.784194 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.784194 2573 reconciler.go:29] "Reconciler: start to sync state" Nov 02 22:47:21.784641 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.784480 2573 factory.go:55] Registering systemd factory Nov 02 22:47:21.784641 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.784518 2573 factory.go:223] Registration of the systemd container factory successfully Nov 02 22:47:21.784798 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.784785 2573 factory.go:153] Registering CRI-O factory Nov 02 22:47:21.784852 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.784800 2573 factory.go:223] Registration of the crio container factory successfully Nov 02 22:47:21.784852 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.784850 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 22:47:21.784949 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.784864 2573 factory.go:103] Registering Raw factory Nov 02 22:47:21.784949 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.784875 2573 manager.go:1199] Started watching for new ooms in manager Nov 02 22:47:21.785646 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.785622 2573 manager.go:319] Starting recovery of all containers Nov 02 22:47:21.797755 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.797740 2573 manager.go:324] Recovery completed Nov 02 22:47:21.801535 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.801517 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:47:21.803373 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.803350 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-176.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:47:21.803454 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.803383 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-176.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:47:21.803454 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.803394 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-176.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:47:21.804058 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.804047 2573 cpu_manager.go:226] "Starting" policy="none" Nov 02 22:47:21.804058 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.804057 2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 22:47:21.804182 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.804074 2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 22:47:21.804182 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.804060 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:47:21.804804 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.804787 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-176.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:47:21.804909 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.804817 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-176.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:47:21.804909 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.804828 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-176.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:47:21.806291 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.806278 2573 policy_none.go:50] "Start" Nov 02 22:47:21.806345 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.806294 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 22:47:21.806345 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.806304 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 22:47:21.808279 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.808269 2573 policy_none.go:44] "Start" Nov 02 22:47:21.841425 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.841388 2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 22:47:21.845459 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.845441 2573 manager.go:341] "Starting Device Plugin manager" Nov 02 22:47:21.845874 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:21.845860 2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 22:47:21.845923 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.845876 2573 server.go:87] "Starting device plugin registration server" Nov 02 22:47:21.846089 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.846077 2573 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 22:47:21.846142 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.846092 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 22:47:21.846283 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.846265 2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 22:47:21.846390 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.846362 2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 22:47:21.846390 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.846380 2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 22:47:21.847042 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:21.847021 2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 22:47:21.847102 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:21.847065 2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-176.us-west-1.compute.internal\" not found" Nov 02 22:47:21.916193 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.916006 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 22:47:21.917498 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.917481 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 22:47:21.917563 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.917506 2573 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 22:47:21.917563 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.917526 2573 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 22:47:21.917563 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:21.917558 2573 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 22:47:21.926562 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.926539 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-176.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 22:47:21.926667 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:21.926539 2573 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-176.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 22:47:21.927337 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:21.926530 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-176.us-west-1.compute.internal.18e45bf0e3f052ef default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-176.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-176.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 22:47:21.774756591 +0000 UTC m=+0.337119919,LastTimestamp:2026-11-02 22:47:21.774756591 +0000 UTC m=+0.337119919,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-176.us-west-1.compute.internal,}" Nov 02 22:47:21.946682 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.946664 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:47:21.947642 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.947624 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-176.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:47:21.947717 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.947657 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-176.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:47:21.947717 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.947670 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-176.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:47:21.947717 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.947694 2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:47:21.998282 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.998215 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 22:47:21.998282 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:21.998269 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kgbkx" Nov 02 22:47:22.018889 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.018858 2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-176.us-west-1.compute.internal"] Nov 02 22:47:22.018989 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.018926 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:47:22.019973 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.019957 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-176.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:47:22.020027 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.019988 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-176.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:47:22.020027 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.020004 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-176.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:47:22.022240 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.022228 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:47:22.022388 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.022371 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:47:22.022427 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.022412 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:47:22.023030 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.023013 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-176.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:47:22.023030 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.023022 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-176.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:47:22.023210 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.023046 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-176.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:47:22.023210 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.023062 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-176.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:47:22.023210 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.023050 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-176.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:47:22.023210 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.023157 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-176.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:47:22.025147 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.025114 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:47:22.025242 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.025161 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:47:22.025871 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.025849 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-176.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:47:22.025954 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.025877 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-176.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:47:22.025954 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.025886 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-176.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:47:22.045451 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:22.045433 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-176.us-west-1.compute.internal\" not found" node="ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:47:22.048234 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:22.048217 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-176.us-west-1.compute.internal\" not found" node="ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:47:22.085005 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.084972 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/049a11ea66e964e484c1cdf0430a7a18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal\" (UID: \"049a11ea66e964e484c1cdf0430a7a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:47:22.085181 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.085036 2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:47:22.183383 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.183358 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:47:22.184553 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.184526 2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:47:22.186976 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.186915 2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:47:22.186976 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:22.186947 2573 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-176.us-west-1.compute.internal\": node \"ip-10-0-135-176.us-west-1.compute.internal\" not found" Nov 02 22:47:22.186976 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.186968 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/049a11ea66e964e484c1cdf0430a7a18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal\" (UID: \"049a11ea66e964e484c1cdf0430a7a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:47:22.187196 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.187006 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/38d433b21a6573d8bddad58877b72617-config\") pod \"kube-apiserver-proxy-ip-10-0-135-176.us-west-1.compute.internal\" (UID: \"38d433b21a6573d8bddad58877b72617\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:47:22.187196 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.187068 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/049a11ea66e964e484c1cdf0430a7a18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal\" (UID: \"049a11ea66e964e484c1cdf0430a7a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:47:22.187196 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.187121 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/049a11ea66e964e484c1cdf0430a7a18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal\" (UID: \"049a11ea66e964e484c1cdf0430a7a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:47:22.187554 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.187534 2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:47:22.284564 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.284481 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:47:22.287630 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.287607 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/38d433b21a6573d8bddad58877b72617-config\") pod \"kube-apiserver-proxy-ip-10-0-135-176.us-west-1.compute.internal\" (UID: \"38d433b21a6573d8bddad58877b72617\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:47:22.287699 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.287641 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/049a11ea66e964e484c1cdf0430a7a18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal\" (UID: \"049a11ea66e964e484c1cdf0430a7a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:47:22.287741 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.287697 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/049a11ea66e964e484c1cdf0430a7a18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal\" (UID: \"049a11ea66e964e484c1cdf0430a7a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:47:22.287741 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.287719 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/38d433b21a6573d8bddad58877b72617-config\") pod \"kube-apiserver-proxy-ip-10-0-135-176.us-west-1.compute.internal\" (UID: \"38d433b21a6573d8bddad58877b72617\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:47:22.348359 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.348323 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:47:22.351297 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.351277 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:47:22.407227 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.407204 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:47:22.407351 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.407314 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:47:22.464845 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.464814 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:47:22.724686 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.724655 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 22:47:22.725258 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.724849 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 22:47:22.725258 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.724849 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 22:47:22.725258 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.724873 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 22:47:22.725258 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:22.724848 2573 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ae6c114d507d041eea76c03a1ba35e05-cac881102500eba0.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-135-176.us-west-1.compute.internal.18e45bf0e5a55c6a\": read tcp 10.0.135.176:58546->77.113.138.105:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-135-176.us-west-1.compute.internal.18e45bf0e5a55c6a default 4804 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-176.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-176.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-176.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 22:47:21 +0000 UTC,LastTimestamp:2026-11-02 22:47:21.947674242 +0000 UTC m=+0.510037570,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-176.us-west-1.compute.internal,}" Nov 02 22:47:22.763557 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.763521 2573 apiserver.go:52] "Watching apiserver" Nov 02 22:47:22.782914 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:22.782894 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 22:47:23.002916 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.002813 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 22:42:22 +0000 UTC" deadline="2028-04-24 13:35:13.094486739 +0000 UTC" Nov 02 22:47:23.002916 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.002863 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12926h47m50.091627424s" Nov 02 22:47:23.033057 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.033026 2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 22:47:23.033416 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.033395 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-176.us-west-1.compute.internal","openshift-network-diagnostics/network-check-target-2qpkq","openshift-ovn-kubernetes/ovnkube-node-pl8m7","kube-system/global-pull-secret-syncer-wbxxp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd","openshift-dns/node-resolver-lhbwd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-t9rnl","openshift-network-operator/iptables-alerter-rw8v4","openshift-multus/multus-mcvpd","openshift-multus/network-metrics-daemon-465bf","openshift-cluster-node-tuning-operator/tuned-2kt76","openshift-image-registry/node-ca-mp88w","kube-system/konnectivity-agent-r5lnw"] Nov 02 22:47:23.035805 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.035786 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qpkq" Nov 02 22:47:23.035897 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:23.035846 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qpkq" podUID="566b2dae-dd3e-4c63-b4d2-e30667a5fd07" Nov 02 22:47:23.037903 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.037887 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.039933 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.039914 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:47:23.040025 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:23.039972 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbxxp" podUID="417d26e1-2a9d-41e3-8940-69626662a2f3" Nov 02 22:47:23.041992 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.041977 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:47:23.044139 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.044108 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lhbwd" Nov 02 22:47:23.046215 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.046199 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:47:23.048458 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.048442 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rw8v4" Nov 02 22:47:23.050686 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.050666 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.052995 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.052982 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:47:23.053063 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:23.053047 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-465bf" podUID="82b3b5e2-09f1-49c0-8d72-c85187918223" Nov 02 22:47:23.055334 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.055321 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.057459 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.057443 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mp88w" Nov 02 22:47:23.059680 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.059662 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r5lnw" Nov 02 22:47:23.085033 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.085017 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 22:47:23.089670 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.089653 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 22:47:23.090036 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.090019 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 22:47:23.090036 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.090023 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mb4zk\"" Nov 02 22:47:23.090184 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.090166 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 22:47:23.090221 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.090187 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 22:47:23.090221 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.090212 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 22:47:23.090287 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.090213 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 22:47:23.091237 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.091223 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-slash\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.091279 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.091244 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-run\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.091279 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.091261 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/457675c6-c725-44e5-a857-97aa4c17e655-cni-binary-copy\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:47:23.091350 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.091277 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-var-lib-cni-multus\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.091350 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.091293 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-etc-openvswitch\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.091426 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.091335 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-run-ovn-kubernetes\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.091426 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.091368 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-cni-netd\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.091426 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.091421 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/472c7524-6a70-420c-aa56-7f3028a53d52-iptables-alerter-script\") pod \"iptables-alerter-rw8v4\" (UID: \"472c7524-6a70-420c-aa56-7f3028a53d52\") " pod="openshift-network-operator/iptables-alerter-rw8v4" Nov 02 22:47:23.091521 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.091446 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82b3b5e2-09f1-49c0-8d72-c85187918223-metrics-certs\") pod \"network-metrics-daemon-465bf\" (UID: \"82b3b5e2-09f1-49c0-8d72-c85187918223\") " pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:47:23.091521 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.091464 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/457675c6-c725-44e5-a857-97aa4c17e655-system-cni-dir\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:47:23.091521 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.091495 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26b7328d-dacf-44c2-995b-306e3ef9cf31-cni-binary-copy\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.091616 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.091524 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-var-lib-cni-bin\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.091616 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.091543 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-kubelet\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.091616 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.091566 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-run-openvswitch\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.091616 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.091588 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.091616 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.091611 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-sysconfig\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.091845 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.091643 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-multus-socket-dir-parent\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.091845 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.091703 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0ea8d3ef-20e9-4ecb-9e50-2df15d7da726-agent-certs\") pod \"konnectivity-agent-r5lnw\" (UID: \"0ea8d3ef-20e9-4ecb-9e50-2df15d7da726\") " pod="kube-system/konnectivity-agent-r5lnw" Nov 02 22:47:23.091845 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.091732 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-var-lib-openvswitch\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.091845 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.091748 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-systemd\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.091845 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.091775 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/417d26e1-2a9d-41e3-8940-69626662a2f3-original-pull-secret\") pod \"global-pull-secret-syncer-wbxxp\" (UID: \"417d26e1-2a9d-41e3-8940-69626662a2f3\") " pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:47:23.091845 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.091803 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-etc-selinux\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:47:23.091845 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.091828 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfpxg\" (UniqueName: \"kubernetes.io/projected/fb045f55-e48d-4cac-88cf-90f90c225cbb-kube-api-access-bfpxg\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:47:23.092189 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.091850 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-run-netns\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.092189 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.091874 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-var-lib-kubelet\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.092189 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.091896 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-hostroot\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.092189 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.091921 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-node-log\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.092189 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.091943 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-log-socket\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.092189 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.091965 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb204d6b-3758-4db4-bd68-0eacec87b805-hosts-file\") pod \"node-resolver-lhbwd\" (UID: \"cb204d6b-3758-4db4-bd68-0eacec87b805\") " pod="openshift-dns/node-resolver-lhbwd" Nov 02 22:47:23.092189 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.091989 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht6kh\" (UniqueName: \"kubernetes.io/projected/13a7b3bb-060c-4383-a864-0575834b221e-kube-api-access-ht6kh\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.092189 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092031 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-registration-dir\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:47:23.092189 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092075 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-run-multus-certs\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.092189 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092112 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-run-systemd\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.092189 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092162 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7321845c-9a95-415b-b6db-2bf983d77fa5-ovn-node-metrics-cert\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.092673 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092203 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/417d26e1-2a9d-41e3-8940-69626662a2f3-dbus\") pod \"global-pull-secret-syncer-wbxxp\" (UID: \"417d26e1-2a9d-41e3-8940-69626662a2f3\") " pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:47:23.092673 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092228 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gzjx\" (UniqueName: \"kubernetes.io/projected/457675c6-c725-44e5-a857-97aa4c17e655-kube-api-access-9gzjx\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:47:23.092673 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092253 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-system-cni-dir\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.092673 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092285 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-cnibin\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.092673 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092314 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-run-k8s-cni-cncf-io\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.092673 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092339 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-etc-kubernetes\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.092673 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092362 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqn4t\" (UniqueName: \"kubernetes.io/projected/566b2dae-dd3e-4c63-b4d2-e30667a5fd07-kube-api-access-cqn4t\") pod \"network-check-target-2qpkq\" (UID: \"566b2dae-dd3e-4c63-b4d2-e30667a5fd07\") " pod="openshift-network-diagnostics/network-check-target-2qpkq" Nov 02 22:47:23.092673 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092389 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njqr9\" (UniqueName: \"kubernetes.io/projected/cb204d6b-3758-4db4-bd68-0eacec87b805-kube-api-access-njqr9\") pod \"node-resolver-lhbwd\" (UID: \"cb204d6b-3758-4db4-bd68-0eacec87b805\") " pod="openshift-dns/node-resolver-lhbwd" Nov 02 22:47:23.092673 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092418 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9bfd\" (UniqueName: \"kubernetes.io/projected/472c7524-6a70-420c-aa56-7f3028a53d52-kube-api-access-l9bfd\") pod \"iptables-alerter-rw8v4\" (UID: \"472c7524-6a70-420c-aa56-7f3028a53d52\") " pod="openshift-network-operator/iptables-alerter-rw8v4" Nov 02 22:47:23.092673 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092439 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/417d26e1-2a9d-41e3-8940-69626662a2f3-kubelet-config\") pod \"global-pull-secret-syncer-wbxxp\" (UID: \"417d26e1-2a9d-41e3-8940-69626662a2f3\") " pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:47:23.092673 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092455 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/472c7524-6a70-420c-aa56-7f3028a53d52-host-slash\") pod \"iptables-alerter-rw8v4\" (UID: \"472c7524-6a70-420c-aa56-7f3028a53d52\") " pod="openshift-network-operator/iptables-alerter-rw8v4" Nov 02 22:47:23.092673 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092469 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-sysctl-d\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.092673 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092494 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zplm\" (UniqueName: \"kubernetes.io/projected/26b7328d-dacf-44c2-995b-306e3ef9cf31-kube-api-access-4zplm\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.092673 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092545 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0ea8d3ef-20e9-4ecb-9e50-2df15d7da726-konnectivity-ca\") pod \"konnectivity-agent-r5lnw\" (UID: \"0ea8d3ef-20e9-4ecb-9e50-2df15d7da726\") " pod="kube-system/konnectivity-agent-r5lnw" Nov 02 22:47:23.092673 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092574 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/457675c6-c725-44e5-a857-97aa4c17e655-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:47:23.092673 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092597 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-multus-cni-dir\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.093401 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092620 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc560bbe-b31a-48e8-8416-dda72f49dae1-host\") pod \"node-ca-mp88w\" (UID: \"fc560bbe-b31a-48e8-8416-dda72f49dae1\") " pod="openshift-image-registry/node-ca-mp88w" Nov 02 22:47:23.093401 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092649 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-cni-bin\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.093401 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092674 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-socket-dir\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:47:23.093401 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092697 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-device-dir\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:47:23.093401 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092719 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-multus-conf-dir\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.093401 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092735 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26b7328d-dacf-44c2-995b-306e3ef9cf31-multus-daemon-config\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.093401 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092748 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-systemd-units\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.093401 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092762 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-run-netns\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.093401 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092784 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t56h\" (UniqueName: \"kubernetes.io/projected/7321845c-9a95-415b-b6db-2bf983d77fa5-kube-api-access-2t56h\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.093401 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092800 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-sysctl-conf\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.093401 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092814 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-lib-modules\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.093401 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092843 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/457675c6-c725-44e5-a857-97aa4c17e655-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:47:23.093401 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092869 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t2t2\" (UniqueName: \"kubernetes.io/projected/fc560bbe-b31a-48e8-8416-dda72f49dae1-kube-api-access-5t2t2\") pod \"node-ca-mp88w\" (UID: \"fc560bbe-b31a-48e8-8416-dda72f49dae1\") " pod="openshift-image-registry/node-ca-mp88w" Nov 02 22:47:23.093401 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092905 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-run-ovn\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.093401 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092928 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7321845c-9a95-415b-b6db-2bf983d77fa5-env-overrides\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.093401 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092947 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-kubernetes\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.093968 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092962 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-var-lib-kubelet\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.093968 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092979 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7321845c-9a95-415b-b6db-2bf983d77fa5-ovnkube-script-lib\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.093968 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.092999 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb204d6b-3758-4db4-bd68-0eacec87b805-tmp-dir\") pod \"node-resolver-lhbwd\" (UID: \"cb204d6b-3758-4db4-bd68-0eacec87b805\") " pod="openshift-dns/node-resolver-lhbwd" Nov 02 22:47:23.093968 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.093014 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-modprobe-d\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.093968 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.093034 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-host\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.093968 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.093059 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/13a7b3bb-060c-4383-a864-0575834b221e-etc-tuned\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.093968 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.093082 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13a7b3bb-060c-4383-a864-0575834b221e-tmp\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.093968 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.093104 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-sys-fs\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:47:23.093968 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.093147 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7321845c-9a95-415b-b6db-2bf983d77fa5-ovnkube-config\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.093968 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.093174 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64dcq\" (UniqueName: \"kubernetes.io/projected/82b3b5e2-09f1-49c0-8d72-c85187918223-kube-api-access-64dcq\") pod \"network-metrics-daemon-465bf\" (UID: \"82b3b5e2-09f1-49c0-8d72-c85187918223\") " pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:47:23.093968 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.093198 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-sys\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.093968 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.093246 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/457675c6-c725-44e5-a857-97aa4c17e655-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:47:23.093968 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.093269 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fc560bbe-b31a-48e8-8416-dda72f49dae1-serviceca\") pod \"node-ca-mp88w\" (UID: \"fc560bbe-b31a-48e8-8416-dda72f49dae1\") " pod="openshift-image-registry/node-ca-mp88w" Nov 02 22:47:23.093968 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.093296 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:47:23.093968 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.093320 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/457675c6-c725-44e5-a857-97aa4c17e655-cnibin\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:47:23.093968 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.093498 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 22:47:23.093968 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.093499 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 22:47:23.093968 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.093538 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 22:47:23.094472 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.093762 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bxrdd\"" Nov 02 22:47:23.095507 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.095493 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f4f84\"" Nov 02 22:47:23.095563 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.095517 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 22:47:23.095563 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.095521 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 22:47:23.097721 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.097701 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 22:47:23.098102 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.098079 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 22:47:23.098211 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.098175 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 22:47:23.098211 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.098194 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hnmtx\"" Nov 02 22:47:23.098301 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.098224 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 22:47:23.098301 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.098237 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 22:47:23.099906 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.099892 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 22:47:23.134454 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.134434 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 22:47:23.134454 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.134450 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 22:47:23.139219 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.139198 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:47:23.139317 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.139260 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-d58gp\"" Nov 02 22:47:23.139769 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.139754 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 22:47:23.139839 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.139802 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 22:47:23.139839 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.139823 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9jqgm\"" Nov 02 22:47:23.139839 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.139831 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 22:47:23.139839 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.139838 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-77tzt\"" Nov 02 22:47:23.139987 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.139823 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:47:23.139987 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.139831 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 22:47:23.143065 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.143050 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-pl59x\"" Nov 02 22:47:23.143358 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.143339 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 22:47:23.143430 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.143363 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p2dkz\"" Nov 02 22:47:23.143430 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.143418 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 22:47:23.145195 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.145175 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 22:47:23.145271 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.145249 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zgcbw" Nov 02 22:47:23.193661 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.193633 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-log-socket\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.193661 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.193664 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb204d6b-3758-4db4-bd68-0eacec87b805-hosts-file\") pod \"node-resolver-lhbwd\" (UID: \"cb204d6b-3758-4db4-bd68-0eacec87b805\") " pod="openshift-dns/node-resolver-lhbwd" Nov 02 22:47:23.193816 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.193686 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht6kh\" (UniqueName: \"kubernetes.io/projected/13a7b3bb-060c-4383-a864-0575834b221e-kube-api-access-ht6kh\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.193816 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.193711 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-registration-dir\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:47:23.193816 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.193721 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-log-socket\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.193816 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.193734 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-run-multus-certs\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.193816 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.193757 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-run-systemd\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.193816 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.193786 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7321845c-9a95-415b-b6db-2bf983d77fa5-ovn-node-metrics-cert\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.193816 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.193808 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-registration-dir\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:47:23.194069 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.193812 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/417d26e1-2a9d-41e3-8940-69626662a2f3-dbus\") pod \"global-pull-secret-syncer-wbxxp\" (UID: \"417d26e1-2a9d-41e3-8940-69626662a2f3\") " pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:47:23.194069 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.193848 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gzjx\" (UniqueName: \"kubernetes.io/projected/457675c6-c725-44e5-a857-97aa4c17e655-kube-api-access-9gzjx\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:47:23.194069 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.193869 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-system-cni-dir\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.194069 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.193889 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-cnibin\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.194069 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.193891 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-run-systemd\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.194069 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.193938 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/417d26e1-2a9d-41e3-8940-69626662a2f3-dbus\") pod \"global-pull-secret-syncer-wbxxp\" (UID: \"417d26e1-2a9d-41e3-8940-69626662a2f3\") " pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:47:23.194069 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.193975 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-system-cni-dir\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.194069 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194039 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-cnibin\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.194407 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194207 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb204d6b-3758-4db4-bd68-0eacec87b805-hosts-file\") pod \"node-resolver-lhbwd\" (UID: \"cb204d6b-3758-4db4-bd68-0eacec87b805\") " pod="openshift-dns/node-resolver-lhbwd" Nov 02 22:47:23.194407 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194216 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-run-k8s-cni-cncf-io\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.194407 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194243 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-etc-kubernetes\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.194407 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194254 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-run-k8s-cni-cncf-io\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.194407 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194264 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqn4t\" (UniqueName: \"kubernetes.io/projected/566b2dae-dd3e-4c63-b4d2-e30667a5fd07-kube-api-access-cqn4t\") pod \"network-check-target-2qpkq\" (UID: \"566b2dae-dd3e-4c63-b4d2-e30667a5fd07\") " pod="openshift-network-diagnostics/network-check-target-2qpkq" Nov 02 22:47:23.194407 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194289 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njqr9\" (UniqueName: \"kubernetes.io/projected/cb204d6b-3758-4db4-bd68-0eacec87b805-kube-api-access-njqr9\") pod \"node-resolver-lhbwd\" (UID: \"cb204d6b-3758-4db4-bd68-0eacec87b805\") " pod="openshift-dns/node-resolver-lhbwd" Nov 02 22:47:23.194407 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194294 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-etc-kubernetes\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.194407 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194388 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9bfd\" (UniqueName: \"kubernetes.io/projected/472c7524-6a70-420c-aa56-7f3028a53d52-kube-api-access-l9bfd\") pod \"iptables-alerter-rw8v4\" (UID: \"472c7524-6a70-420c-aa56-7f3028a53d52\") " pod="openshift-network-operator/iptables-alerter-rw8v4" Nov 02 22:47:23.194762 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194416 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/417d26e1-2a9d-41e3-8940-69626662a2f3-kubelet-config\") pod \"global-pull-secret-syncer-wbxxp\" (UID: \"417d26e1-2a9d-41e3-8940-69626662a2f3\") " pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:47:23.194762 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194443 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/472c7524-6a70-420c-aa56-7f3028a53d52-host-slash\") pod \"iptables-alerter-rw8v4\" (UID: \"472c7524-6a70-420c-aa56-7f3028a53d52\") " pod="openshift-network-operator/iptables-alerter-rw8v4" Nov 02 22:47:23.194762 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194464 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-sysctl-d\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.194762 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194491 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zplm\" (UniqueName: \"kubernetes.io/projected/26b7328d-dacf-44c2-995b-306e3ef9cf31-kube-api-access-4zplm\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.194762 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194505 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/417d26e1-2a9d-41e3-8940-69626662a2f3-kubelet-config\") pod \"global-pull-secret-syncer-wbxxp\" (UID: \"417d26e1-2a9d-41e3-8940-69626662a2f3\") " pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:47:23.194762 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194518 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0ea8d3ef-20e9-4ecb-9e50-2df15d7da726-konnectivity-ca\") pod \"konnectivity-agent-r5lnw\" (UID: \"0ea8d3ef-20e9-4ecb-9e50-2df15d7da726\") " pod="kube-system/konnectivity-agent-r5lnw" Nov 02 22:47:23.194762 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194540 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/472c7524-6a70-420c-aa56-7f3028a53d52-host-slash\") pod \"iptables-alerter-rw8v4\" (UID: \"472c7524-6a70-420c-aa56-7f3028a53d52\") " pod="openshift-network-operator/iptables-alerter-rw8v4" Nov 02 22:47:23.194762 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194545 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/457675c6-c725-44e5-a857-97aa4c17e655-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:47:23.194762 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194554 2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 22:47:23.194762 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194571 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-multus-cni-dir\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.194762 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194594 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc560bbe-b31a-48e8-8416-dda72f49dae1-host\") pod \"node-ca-mp88w\" (UID: \"fc560bbe-b31a-48e8-8416-dda72f49dae1\") " pod="openshift-image-registry/node-ca-mp88w" Nov 02 22:47:23.194762 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194615 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-cni-bin\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.194762 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194612 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-sysctl-d\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.194762 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194654 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc560bbe-b31a-48e8-8416-dda72f49dae1-host\") pod \"node-ca-mp88w\" (UID: \"fc560bbe-b31a-48e8-8416-dda72f49dae1\") " pod="openshift-image-registry/node-ca-mp88w" Nov 02 22:47:23.194762 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194660 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-multus-cni-dir\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.194762 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194677 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-cni-bin\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.194762 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194681 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-socket-dir\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:47:23.194762 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194726 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-device-dir\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:47:23.197604 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194747 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-multus-conf-dir\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.197604 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194799 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26b7328d-dacf-44c2-995b-306e3ef9cf31-multus-daemon-config\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.197604 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194825 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-systemd-units\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.197604 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194840 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-multus-conf-dir\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.197604 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194800 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-device-dir\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:47:23.197604 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194840 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-socket-dir\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:47:23.197604 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194848 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-run-netns\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.197604 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194889 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-systemd-units\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.197604 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194925 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t56h\" (UniqueName: \"kubernetes.io/projected/7321845c-9a95-415b-b6db-2bf983d77fa5-kube-api-access-2t56h\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.197604 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194979 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-run-netns\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.197604 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195012 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-sysctl-conf\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.197604 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195049 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-lib-modules\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.197604 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195091 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/457675c6-c725-44e5-a857-97aa4c17e655-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:47:23.197604 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195119 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t2t2\" (UniqueName: \"kubernetes.io/projected/fc560bbe-b31a-48e8-8416-dda72f49dae1-kube-api-access-5t2t2\") pod \"node-ca-mp88w\" (UID: \"fc560bbe-b31a-48e8-8416-dda72f49dae1\") " pod="openshift-image-registry/node-ca-mp88w" Nov 02 22:47:23.197604 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195121 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/457675c6-c725-44e5-a857-97aa4c17e655-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:47:23.197604 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195161 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-run-ovn\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.197604 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195175 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-lib-modules\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.198366 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195091 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0ea8d3ef-20e9-4ecb-9e50-2df15d7da726-konnectivity-ca\") pod \"konnectivity-agent-r5lnw\" (UID: \"0ea8d3ef-20e9-4ecb-9e50-2df15d7da726\") " pod="kube-system/konnectivity-agent-r5lnw" Nov 02 22:47:23.198366 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195195 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-sysctl-conf\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.198366 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195238 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7321845c-9a95-415b-b6db-2bf983d77fa5-env-overrides\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.198366 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195250 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-run-ovn\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.198366 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195249 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/457675c6-c725-44e5-a857-97aa4c17e655-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:47:23.198366 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195287 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-kubernetes\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.198366 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195318 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-kubernetes\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.198366 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195329 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-var-lib-kubelet\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.198366 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.194235 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-run-multus-certs\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.198366 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195378 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7321845c-9a95-415b-b6db-2bf983d77fa5-ovnkube-script-lib\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.198366 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195403 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb204d6b-3758-4db4-bd68-0eacec87b805-tmp-dir\") pod \"node-resolver-lhbwd\" (UID: \"cb204d6b-3758-4db4-bd68-0eacec87b805\") " pod="openshift-dns/node-resolver-lhbwd" Nov 02 22:47:23.198366 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195456 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26b7328d-dacf-44c2-995b-306e3ef9cf31-multus-daemon-config\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.198366 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195457 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-modprobe-d\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.198366 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195497 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-host\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.198366 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195516 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/13a7b3bb-060c-4383-a864-0575834b221e-etc-tuned\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.198366 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195529 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13a7b3bb-060c-4383-a864-0575834b221e-tmp\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.198366 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195555 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-host\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.198366 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195565 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-sys-fs\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:47:23.199225 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195577 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-modprobe-d\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.199225 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195602 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7321845c-9a95-415b-b6db-2bf983d77fa5-ovnkube-config\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.199225 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195633 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64dcq\" (UniqueName: \"kubernetes.io/projected/82b3b5e2-09f1-49c0-8d72-c85187918223-kube-api-access-64dcq\") pod \"network-metrics-daemon-465bf\" (UID: \"82b3b5e2-09f1-49c0-8d72-c85187918223\") " pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:47:23.199225 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195644 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb204d6b-3758-4db4-bd68-0eacec87b805-tmp-dir\") pod \"node-resolver-lhbwd\" (UID: \"cb204d6b-3758-4db4-bd68-0eacec87b805\") " pod="openshift-dns/node-resolver-lhbwd" Nov 02 22:47:23.199225 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195656 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-sys\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.199225 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195608 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-sys-fs\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:47:23.199225 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195683 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/457675c6-c725-44e5-a857-97aa4c17e655-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:47:23.199225 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195705 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fc560bbe-b31a-48e8-8416-dda72f49dae1-serviceca\") pod \"node-ca-mp88w\" (UID: \"fc560bbe-b31a-48e8-8416-dda72f49dae1\") " pod="openshift-image-registry/node-ca-mp88w" Nov 02 22:47:23.199225 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195728 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:47:23.199225 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195752 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/457675c6-c725-44e5-a857-97aa4c17e655-cnibin\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:47:23.199225 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195776 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-slash\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.199225 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195798 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-run\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.199225 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195843 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/457675c6-c725-44e5-a857-97aa4c17e655-cni-binary-copy\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:47:23.199225 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195866 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-var-lib-cni-multus\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.199225 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195889 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-etc-openvswitch\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.199225 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195456 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-var-lib-kubelet\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.199225 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195912 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-run-ovn-kubernetes\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.200113 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195929 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-slash\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.200113 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195938 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-cni-netd\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.200113 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195961 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/472c7524-6a70-420c-aa56-7f3028a53d52-iptables-alerter-script\") pod \"iptables-alerter-rw8v4\" (UID: \"472c7524-6a70-420c-aa56-7f3028a53d52\") " pod="openshift-network-operator/iptables-alerter-rw8v4" Nov 02 22:47:23.200113 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195986 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82b3b5e2-09f1-49c0-8d72-c85187918223-metrics-certs\") pod \"network-metrics-daemon-465bf\" (UID: \"82b3b5e2-09f1-49c0-8d72-c85187918223\") " pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:47:23.200113 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196011 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/457675c6-c725-44e5-a857-97aa4c17e655-system-cni-dir\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:47:23.200113 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196035 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26b7328d-dacf-44c2-995b-306e3ef9cf31-cni-binary-copy\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.200113 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196059 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-var-lib-cni-bin\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.200113 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196081 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-kubelet\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.200113 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196105 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-run-openvswitch\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.200113 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196156 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.200113 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196183 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-sysconfig\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.200113 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196205 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-multus-socket-dir-parent\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.200113 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196229 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0ea8d3ef-20e9-4ecb-9e50-2df15d7da726-agent-certs\") pod \"konnectivity-agent-r5lnw\" (UID: \"0ea8d3ef-20e9-4ecb-9e50-2df15d7da726\") " pod="kube-system/konnectivity-agent-r5lnw" Nov 02 22:47:23.200113 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196289 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-var-lib-openvswitch\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.200113 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196315 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-systemd\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.200113 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196318 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7321845c-9a95-415b-b6db-2bf983d77fa5-env-overrides\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.200113 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196326 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:47:23.200918 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196343 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/417d26e1-2a9d-41e3-8940-69626662a2f3-original-pull-secret\") pod \"global-pull-secret-syncer-wbxxp\" (UID: \"417d26e1-2a9d-41e3-8940-69626662a2f3\") " pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:47:23.200918 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196371 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-etc-selinux\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:47:23.200918 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196385 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/457675c6-c725-44e5-a857-97aa4c17e655-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:47:23.200918 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196397 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfpxg\" (UniqueName: \"kubernetes.io/projected/fb045f55-e48d-4cac-88cf-90f90c225cbb-kube-api-access-bfpxg\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:47:23.200918 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196386 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/457675c6-c725-44e5-a857-97aa4c17e655-cnibin\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:47:23.200918 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196427 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-run-netns\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.200918 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196431 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-var-lib-cni-bin\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.200918 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196452 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-var-lib-kubelet\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.200918 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196455 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-run-ovn-kubernetes\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.200918 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196478 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-hostroot\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.200918 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196504 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-node-log\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.200918 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196504 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-run\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.200918 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196592 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-kubelet\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.200918 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.195753 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-sys\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.200918 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196637 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-run-openvswitch\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.200918 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196721 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-etc-selinux\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:47:23.200918 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196798 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fc560bbe-b31a-48e8-8416-dda72f49dae1-serviceca\") pod \"node-ca-mp88w\" (UID: \"fc560bbe-b31a-48e8-8416-dda72f49dae1\") " pod="openshift-image-registry/node-ca-mp88w" Nov 02 22:47:23.200918 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196831 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-run-netns\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.201807 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196478 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-cni-netd\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.201807 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196800 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.201807 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196891 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-multus-socket-dir-parent\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.201807 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196931 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/457675c6-c725-44e5-a857-97aa4c17e655-cni-binary-copy\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:47:23.201807 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196931 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-sysconfig\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.201807 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196958 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/457675c6-c725-44e5-a857-97aa4c17e655-system-cni-dir\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:47:23.201807 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.196968 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-systemd\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.201807 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.197000 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-var-lib-kubelet\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.201807 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.197013 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/472c7524-6a70-420c-aa56-7f3028a53d52-iptables-alerter-script\") pod \"iptables-alerter-rw8v4\" (UID: \"472c7524-6a70-420c-aa56-7f3028a53d52\") " pod="openshift-network-operator/iptables-alerter-rw8v4" Nov 02 22:47:23.201807 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.197023 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-var-lib-openvswitch\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.201807 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:23.197665 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:23.201807 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:23.197764 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82b3b5e2-09f1-49c0-8d72-c85187918223-metrics-certs podName:82b3b5e2-09f1-49c0-8d72-c85187918223 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:23.697727303 +0000 UTC m=+2.260090638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82b3b5e2-09f1-49c0-8d72-c85187918223-metrics-certs") pod "network-metrics-daemon-465bf" (UID: "82b3b5e2-09f1-49c0-8d72-c85187918223") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:23.201807 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.197031 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-hostroot\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.201807 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:23.198476 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:23.201807 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.198546 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-node-log\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.201807 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.198571 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-var-lib-cni-multus\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.201807 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:23.198613 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/417d26e1-2a9d-41e3-8940-69626662a2f3-original-pull-secret podName:417d26e1-2a9d-41e3-8940-69626662a2f3 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:23.698588033 +0000 UTC m=+2.260951363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/417d26e1-2a9d-41e3-8940-69626662a2f3-original-pull-secret") pod "global-pull-secret-syncer-wbxxp" (UID: "417d26e1-2a9d-41e3-8940-69626662a2f3") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:23.202630 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.198669 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-etc-openvswitch\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.202630 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.198878 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13a7b3bb-060c-4383-a864-0575834b221e-tmp\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.202630 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.200637 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26b7328d-dacf-44c2-995b-306e3ef9cf31-cni-binary-copy\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.202630 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.200713 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/13a7b3bb-060c-4383-a864-0575834b221e-etc-tuned\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.202630 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.200754 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7321845c-9a95-415b-b6db-2bf983d77fa5-ovn-node-metrics-cert\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.202630 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.200842 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0ea8d3ef-20e9-4ecb-9e50-2df15d7da726-agent-certs\") pod \"konnectivity-agent-r5lnw\" (UID: \"0ea8d3ef-20e9-4ecb-9e50-2df15d7da726\") " pod="kube-system/konnectivity-agent-r5lnw" Nov 02 22:47:23.202630 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.200912 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7321845c-9a95-415b-b6db-2bf983d77fa5-ovnkube-config\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.202630 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.202319 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7321845c-9a95-415b-b6db-2bf983d77fa5-ovnkube-script-lib\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.253177 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:23.253108 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:47:23.253177 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:23.253147 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:47:23.253177 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:23.253159 2573 projected.go:196] Error preparing data for projected volume kube-api-access-cqn4t for pod openshift-network-diagnostics/network-check-target-2qpkq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:23.253360 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:23.253219 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/566b2dae-dd3e-4c63-b4d2-e30667a5fd07-kube-api-access-cqn4t podName:566b2dae-dd3e-4c63-b4d2-e30667a5fd07 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:23.753202157 +0000 UTC m=+2.315565489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cqn4t" (UniqueName: "kubernetes.io/projected/566b2dae-dd3e-4c63-b4d2-e30667a5fd07-kube-api-access-cqn4t") pod "network-check-target-2qpkq" (UID: "566b2dae-dd3e-4c63-b4d2-e30667a5fd07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:23.255944 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.255910 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht6kh\" (UniqueName: \"kubernetes.io/projected/13a7b3bb-060c-4383-a864-0575834b221e-kube-api-access-ht6kh\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.257372 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.257313 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njqr9\" (UniqueName: \"kubernetes.io/projected/cb204d6b-3758-4db4-bd68-0eacec87b805-kube-api-access-njqr9\") pod \"node-resolver-lhbwd\" (UID: \"cb204d6b-3758-4db4-bd68-0eacec87b805\") " pod="openshift-dns/node-resolver-lhbwd" Nov 02 22:47:23.257955 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.257908 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zplm\" (UniqueName: \"kubernetes.io/projected/26b7328d-dacf-44c2-995b-306e3ef9cf31-kube-api-access-4zplm\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.258045 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.257972 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfpxg\" (UniqueName: \"kubernetes.io/projected/fb045f55-e48d-4cac-88cf-90f90c225cbb-kube-api-access-bfpxg\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:47:23.258045 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.257986 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9bfd\" (UniqueName: \"kubernetes.io/projected/472c7524-6a70-420c-aa56-7f3028a53d52-kube-api-access-l9bfd\") pod \"iptables-alerter-rw8v4\" (UID: \"472c7524-6a70-420c-aa56-7f3028a53d52\") " pod="openshift-network-operator/iptables-alerter-rw8v4" Nov 02 22:47:23.258045 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.257973 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t2t2\" (UniqueName: \"kubernetes.io/projected/fc560bbe-b31a-48e8-8416-dda72f49dae1-kube-api-access-5t2t2\") pod \"node-ca-mp88w\" (UID: \"fc560bbe-b31a-48e8-8416-dda72f49dae1\") " pod="openshift-image-registry/node-ca-mp88w" Nov 02 22:47:23.258218 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.258146 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64dcq\" (UniqueName: \"kubernetes.io/projected/82b3b5e2-09f1-49c0-8d72-c85187918223-kube-api-access-64dcq\") pod \"network-metrics-daemon-465bf\" (UID: \"82b3b5e2-09f1-49c0-8d72-c85187918223\") " pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:47:23.258321 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.258306 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t56h\" (UniqueName: \"kubernetes.io/projected/7321845c-9a95-415b-b6db-2bf983d77fa5-kube-api-access-2t56h\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.259448 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.259431 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gzjx\" (UniqueName: \"kubernetes.io/projected/457675c6-c725-44e5-a857-97aa4c17e655-kube-api-access-9gzjx\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:47:23.385226 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.385183 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:23.392445 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.392423 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:47:23.400395 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.400368 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lhbwd" Nov 02 22:47:23.426974 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.426949 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:47:23.429969 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.429945 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rw8v4" Nov 02 22:47:23.444969 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.444950 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mcvpd" Nov 02 22:47:23.458553 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.458520 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:47:23.472165 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.472062 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mp88w" Nov 02 22:47:23.475708 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.475691 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r5lnw" Nov 02 22:47:23.510431 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:23.510354 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod049a11ea66e964e484c1cdf0430a7a18.slice/crio-2d0f15b28f79d9d966377869a930b9cd17054031a166bfbbd0473f25097a1d2a WatchSource:0}: Error finding container 2d0f15b28f79d9d966377869a930b9cd17054031a166bfbbd0473f25097a1d2a: Status 404 returned error can't find the container with id 2d0f15b28f79d9d966377869a930b9cd17054031a166bfbbd0473f25097a1d2a Nov 02 22:47:23.510807 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:23.510775 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d433b21a6573d8bddad58877b72617.slice/crio-149bfac297d1959995d6a77e8d0cb2e89a62a5efe665a352f8dd48c6a210de40 WatchSource:0}: Error finding container 149bfac297d1959995d6a77e8d0cb2e89a62a5efe665a352f8dd48c6a210de40: Status 404 returned error can't find the container with id 149bfac297d1959995d6a77e8d0cb2e89a62a5efe665a352f8dd48c6a210de40 Nov 02 22:47:23.515376 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.515360 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:47:23.700958 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.700931 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82b3b5e2-09f1-49c0-8d72-c85187918223-metrics-certs\") pod \"network-metrics-daemon-465bf\" (UID: \"82b3b5e2-09f1-49c0-8d72-c85187918223\") " pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:47:23.701090 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.700969 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/417d26e1-2a9d-41e3-8940-69626662a2f3-original-pull-secret\") pod \"global-pull-secret-syncer-wbxxp\" (UID: \"417d26e1-2a9d-41e3-8940-69626662a2f3\") " pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:47:23.701135 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:23.701090 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:23.701135 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:23.701100 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:23.701218 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:23.701154 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/417d26e1-2a9d-41e3-8940-69626662a2f3-original-pull-secret podName:417d26e1-2a9d-41e3-8940-69626662a2f3 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:24.701141766 +0000 UTC m=+3.263505081 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/417d26e1-2a9d-41e3-8940-69626662a2f3-original-pull-secret") pod "global-pull-secret-syncer-wbxxp" (UID: "417d26e1-2a9d-41e3-8940-69626662a2f3") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:23.701218 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:23.701196 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82b3b5e2-09f1-49c0-8d72-c85187918223-metrics-certs podName:82b3b5e2-09f1-49c0-8d72-c85187918223 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:24.701184164 +0000 UTC m=+3.263547482 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82b3b5e2-09f1-49c0-8d72-c85187918223-metrics-certs") pod "network-metrics-daemon-465bf" (UID: "82b3b5e2-09f1-49c0-8d72-c85187918223") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:23.801732 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.801651 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqn4t\" (UniqueName: \"kubernetes.io/projected/566b2dae-dd3e-4c63-b4d2-e30667a5fd07-kube-api-access-cqn4t\") pod \"network-check-target-2qpkq\" (UID: \"566b2dae-dd3e-4c63-b4d2-e30667a5fd07\") " pod="openshift-network-diagnostics/network-check-target-2qpkq" Nov 02 22:47:23.802401 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:23.801767 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:47:23.802401 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:23.801783 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:47:23.802401 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:23.801793 2573 projected.go:196] Error preparing data for projected volume kube-api-access-cqn4t for pod openshift-network-diagnostics/network-check-target-2qpkq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:23.802401 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:23.801841 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/566b2dae-dd3e-4c63-b4d2-e30667a5fd07-kube-api-access-cqn4t podName:566b2dae-dd3e-4c63-b4d2-e30667a5fd07 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:24.801827712 +0000 UTC m=+3.364191042 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cqn4t" (UniqueName: "kubernetes.io/projected/566b2dae-dd3e-4c63-b4d2-e30667a5fd07-kube-api-access-cqn4t") pod "network-check-target-2qpkq" (UID: "566b2dae-dd3e-4c63-b4d2-e30667a5fd07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:23.922013 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.921947 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.us-west-1.compute.internal" event={"ID":"38d433b21a6573d8bddad58877b72617","Type":"ContainerStarted","Data":"149bfac297d1959995d6a77e8d0cb2e89a62a5efe665a352f8dd48c6a210de40"} Nov 02 22:47:23.923154 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.923132 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal" event={"ID":"049a11ea66e964e484c1cdf0430a7a18","Type":"ContainerStarted","Data":"2d0f15b28f79d9d966377869a930b9cd17054031a166bfbbd0473f25097a1d2a"} Nov 02 22:47:23.933744 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:23.933724 2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:47:24.145801 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:24.145773 2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:47:24.147799 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:24.147768 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 22:42:23 +0000 UTC" deadline="2028-05-13 19:53:35.361888742 +0000 UTC" Nov 02 22:47:24.147799 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:24.147793 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13389h6m11.214098848s" Nov 02 22:47:24.238222 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:24.238183 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7321845c_9a95_415b_b6db_2bf983d77fa5.slice/crio-12d0b5f3c4797a1b4b8958ae37822899ad7f836fe7527324eff473eb7eee945b WatchSource:0}: Error finding container 12d0b5f3c4797a1b4b8958ae37822899ad7f836fe7527324eff473eb7eee945b: Status 404 returned error can't find the container with id 12d0b5f3c4797a1b4b8958ae37822899ad7f836fe7527324eff473eb7eee945b Nov 02 22:47:24.241704 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:24.241684 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:47:24.258962 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:24.258938 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb204d6b_3758_4db4_bd68_0eacec87b805.slice/crio-57a51cd7c0976e7981c9f7af8100910430f2502147ec3d8f927b084372b98d6f WatchSource:0}: Error finding container 57a51cd7c0976e7981c9f7af8100910430f2502147ec3d8f927b084372b98d6f: Status 404 returned error can't find the container with id 57a51cd7c0976e7981c9f7af8100910430f2502147ec3d8f927b084372b98d6f Nov 02 22:47:24.279170 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:24.279139 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb045f55_e48d_4cac_88cf_90f90c225cbb.slice/crio-84137200c098651df0db031ffd4f9cda174e1d803d3774c76934ca174437c773 WatchSource:0}: Error finding container 84137200c098651df0db031ffd4f9cda174e1d803d3774c76934ca174437c773: Status 404 returned error can't find the container with id 84137200c098651df0db031ffd4f9cda174e1d803d3774c76934ca174437c773 Nov 02 22:47:24.306818 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:24.306794 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod472c7524_6a70_420c_aa56_7f3028a53d52.slice/crio-20aeb9b1286b40414ad4b557e23c6578bd0da566586adfb27316b45cd1632aef WatchSource:0}: Error finding container 20aeb9b1286b40414ad4b557e23c6578bd0da566586adfb27316b45cd1632aef: Status 404 returned error can't find the container with id 20aeb9b1286b40414ad4b557e23c6578bd0da566586adfb27316b45cd1632aef Nov 02 22:47:24.316924 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:24.316899 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc560bbe_b31a_48e8_8416_dda72f49dae1.slice/crio-c18c7b4723c6ab8f8f18e63600604f43a91a32b392ce37c0254c28b41b6e81dd WatchSource:0}: Error finding container c18c7b4723c6ab8f8f18e63600604f43a91a32b392ce37c0254c28b41b6e81dd: Status 404 returned error can't find the container with id c18c7b4723c6ab8f8f18e63600604f43a91a32b392ce37c0254c28b41b6e81dd Nov 02 22:47:24.349694 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:24.349667 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b7328d_dacf_44c2_995b_306e3ef9cf31.slice/crio-278cf9de5cdac5e81c0f35ddc4d882c8f4fd7f0a7a2cb7f4717d54c016007ec0 WatchSource:0}: Error finding container 278cf9de5cdac5e81c0f35ddc4d882c8f4fd7f0a7a2cb7f4717d54c016007ec0: Status 404 returned error can't find the container with id 278cf9de5cdac5e81c0f35ddc4d882c8f4fd7f0a7a2cb7f4717d54c016007ec0 Nov 02 22:47:24.380074 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:24.380050 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ea8d3ef_20e9_4ecb_9e50_2df15d7da726.slice/crio-5399dea54dbec22dce782d3b4c70b29d1f0dfd0191efc4025948b94dc5a9cbd9 WatchSource:0}: Error finding container 5399dea54dbec22dce782d3b4c70b29d1f0dfd0191efc4025948b94dc5a9cbd9: Status 404 returned error can't find the container with id 5399dea54dbec22dce782d3b4c70b29d1f0dfd0191efc4025948b94dc5a9cbd9 Nov 02 22:47:24.386956 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:24.386929 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a7b3bb_060c_4383_a864_0575834b221e.slice/crio-9c7a8090fcbbe648a32e05785017b55b3341f41f072cff499c217392b2fd959b WatchSource:0}: Error finding container 9c7a8090fcbbe648a32e05785017b55b3341f41f072cff499c217392b2fd959b: Status 404 returned error can't find the container with id 9c7a8090fcbbe648a32e05785017b55b3341f41f072cff499c217392b2fd959b Nov 02 22:47:24.407396 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:24.407358 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod457675c6_c725_44e5_a857_97aa4c17e655.slice/crio-5ed5ea0522eca41e9753ad49cca68438d6ba6ae49d49745bcc1487e938928f84 WatchSource:0}: Error finding container 5ed5ea0522eca41e9753ad49cca68438d6ba6ae49d49745bcc1487e938928f84: Status 404 returned error can't find the container with id 5ed5ea0522eca41e9753ad49cca68438d6ba6ae49d49745bcc1487e938928f84 Nov 02 22:47:24.709989 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:24.709887 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/417d26e1-2a9d-41e3-8940-69626662a2f3-original-pull-secret\") pod \"global-pull-secret-syncer-wbxxp\" (UID: \"417d26e1-2a9d-41e3-8940-69626662a2f3\") " pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:47:24.710155 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:24.709987 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82b3b5e2-09f1-49c0-8d72-c85187918223-metrics-certs\") pod \"network-metrics-daemon-465bf\" (UID: \"82b3b5e2-09f1-49c0-8d72-c85187918223\") " pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:47:24.710155 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:24.710119 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:24.710257 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:24.710199 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82b3b5e2-09f1-49c0-8d72-c85187918223-metrics-certs podName:82b3b5e2-09f1-49c0-8d72-c85187918223 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:26.710163365 +0000 UTC m=+5.272526685 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82b3b5e2-09f1-49c0-8d72-c85187918223-metrics-certs") pod "network-metrics-daemon-465bf" (UID: "82b3b5e2-09f1-49c0-8d72-c85187918223") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:24.710307 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:24.710269 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:24.710307 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:24.710305 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/417d26e1-2a9d-41e3-8940-69626662a2f3-original-pull-secret podName:417d26e1-2a9d-41e3-8940-69626662a2f3 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:26.710293324 +0000 UTC m=+5.272656648 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/417d26e1-2a9d-41e3-8940-69626662a2f3-original-pull-secret") pod "global-pull-secret-syncer-wbxxp" (UID: "417d26e1-2a9d-41e3-8940-69626662a2f3") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:24.810818 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:24.810772 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqn4t\" (UniqueName: \"kubernetes.io/projected/566b2dae-dd3e-4c63-b4d2-e30667a5fd07-kube-api-access-cqn4t\") pod \"network-check-target-2qpkq\" (UID: \"566b2dae-dd3e-4c63-b4d2-e30667a5fd07\") " pod="openshift-network-diagnostics/network-check-target-2qpkq" Nov 02 22:47:24.811494 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:24.810982 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:47:24.811494 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:24.811012 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:47:24.811494 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:24.811027 2573 projected.go:196] Error preparing data for projected volume kube-api-access-cqn4t for pod openshift-network-diagnostics/network-check-target-2qpkq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:24.811494 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:24.811106 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/566b2dae-dd3e-4c63-b4d2-e30667a5fd07-kube-api-access-cqn4t podName:566b2dae-dd3e-4c63-b4d2-e30667a5fd07 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:26.81108527 +0000 UTC m=+5.373448594 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cqn4t" (UniqueName: "kubernetes.io/projected/566b2dae-dd3e-4c63-b4d2-e30667a5fd07-kube-api-access-cqn4t") pod "network-check-target-2qpkq" (UID: "566b2dae-dd3e-4c63-b4d2-e30667a5fd07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:24.918887 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:24.918847 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:47:24.918887 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:24.918869 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:47:24.919115 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:24.918999 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbxxp" podUID="417d26e1-2a9d-41e3-8940-69626662a2f3" Nov 02 22:47:24.919115 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:24.919045 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qpkq" Nov 02 22:47:24.919258 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:24.919120 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-465bf" podUID="82b3b5e2-09f1-49c0-8d72-c85187918223" Nov 02 22:47:24.919258 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:24.919244 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qpkq" podUID="566b2dae-dd3e-4c63-b4d2-e30667a5fd07" Nov 02 22:47:24.926493 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:24.926464 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9rnl" event={"ID":"457675c6-c725-44e5-a857-97aa4c17e655","Type":"ContainerStarted","Data":"5ed5ea0522eca41e9753ad49cca68438d6ba6ae49d49745bcc1487e938928f84"} Nov 02 22:47:24.927831 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:24.927802 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2kt76" event={"ID":"13a7b3bb-060c-4383-a864-0575834b221e","Type":"ContainerStarted","Data":"9c7a8090fcbbe648a32e05785017b55b3341f41f072cff499c217392b2fd959b"} Nov 02 22:47:24.929199 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:24.929174 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r5lnw" event={"ID":"0ea8d3ef-20e9-4ecb-9e50-2df15d7da726","Type":"ContainerStarted","Data":"5399dea54dbec22dce782d3b4c70b29d1f0dfd0191efc4025948b94dc5a9cbd9"} Nov 02 22:47:24.930380 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:24.930351 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mcvpd" event={"ID":"26b7328d-dacf-44c2-995b-306e3ef9cf31","Type":"ContainerStarted","Data":"278cf9de5cdac5e81c0f35ddc4d882c8f4fd7f0a7a2cb7f4717d54c016007ec0"} Nov 02 22:47:24.931515 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:24.931495 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mp88w" event={"ID":"fc560bbe-b31a-48e8-8416-dda72f49dae1","Type":"ContainerStarted","Data":"c18c7b4723c6ab8f8f18e63600604f43a91a32b392ce37c0254c28b41b6e81dd"} Nov 02 22:47:24.932741 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:24.932715 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" event={"ID":"fb045f55-e48d-4cac-88cf-90f90c225cbb","Type":"ContainerStarted","Data":"84137200c098651df0db031ffd4f9cda174e1d803d3774c76934ca174437c773"} Nov 02 22:47:24.934049 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:24.934024 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rw8v4" event={"ID":"472c7524-6a70-420c-aa56-7f3028a53d52","Type":"ContainerStarted","Data":"20aeb9b1286b40414ad4b557e23c6578bd0da566586adfb27316b45cd1632aef"} Nov 02 22:47:24.935327 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:24.935291 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lhbwd" event={"ID":"cb204d6b-3758-4db4-bd68-0eacec87b805","Type":"ContainerStarted","Data":"57a51cd7c0976e7981c9f7af8100910430f2502147ec3d8f927b084372b98d6f"} Nov 02 22:47:24.936326 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:24.936300 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" event={"ID":"7321845c-9a95-415b-b6db-2bf983d77fa5","Type":"ContainerStarted","Data":"12d0b5f3c4797a1b4b8958ae37822899ad7f836fe7527324eff473eb7eee945b"} Nov 02 22:47:25.148636 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:25.148565 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 22:42:23 +0000 UTC" deadline="2028-06-06 20:02:26.988915937 +0000 UTC" Nov 02 22:47:25.148636 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:25.148633 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13965h15m1.840286693s" Nov 02 22:47:26.732576 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:26.732536 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82b3b5e2-09f1-49c0-8d72-c85187918223-metrics-certs\") pod \"network-metrics-daemon-465bf\" (UID: \"82b3b5e2-09f1-49c0-8d72-c85187918223\") " pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:47:26.733111 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:26.732596 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/417d26e1-2a9d-41e3-8940-69626662a2f3-original-pull-secret\") pod \"global-pull-secret-syncer-wbxxp\" (UID: \"417d26e1-2a9d-41e3-8940-69626662a2f3\") " pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:47:26.733111 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:26.732757 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:26.733111 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:26.732822 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/417d26e1-2a9d-41e3-8940-69626662a2f3-original-pull-secret podName:417d26e1-2a9d-41e3-8940-69626662a2f3 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:30.732804361 +0000 UTC m=+9.295167682 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/417d26e1-2a9d-41e3-8940-69626662a2f3-original-pull-secret") pod "global-pull-secret-syncer-wbxxp" (UID: "417d26e1-2a9d-41e3-8940-69626662a2f3") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:26.733111 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:26.732902 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:26.733111 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:26.732945 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82b3b5e2-09f1-49c0-8d72-c85187918223-metrics-certs podName:82b3b5e2-09f1-49c0-8d72-c85187918223 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:30.732932015 +0000 UTC m=+9.295295337 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82b3b5e2-09f1-49c0-8d72-c85187918223-metrics-certs") pod "network-metrics-daemon-465bf" (UID: "82b3b5e2-09f1-49c0-8d72-c85187918223") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:26.833154 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:26.833115 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqn4t\" (UniqueName: \"kubernetes.io/projected/566b2dae-dd3e-4c63-b4d2-e30667a5fd07-kube-api-access-cqn4t\") pod \"network-check-target-2qpkq\" (UID: \"566b2dae-dd3e-4c63-b4d2-e30667a5fd07\") " pod="openshift-network-diagnostics/network-check-target-2qpkq" Nov 02 22:47:26.833345 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:26.833303 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:47:26.833345 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:26.833326 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:47:26.833345 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:26.833338 2573 projected.go:196] Error preparing data for projected volume kube-api-access-cqn4t for pod openshift-network-diagnostics/network-check-target-2qpkq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:26.833517 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:26.833397 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/566b2dae-dd3e-4c63-b4d2-e30667a5fd07-kube-api-access-cqn4t podName:566b2dae-dd3e-4c63-b4d2-e30667a5fd07 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:30.833378088 +0000 UTC m=+9.395741424 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cqn4t" (UniqueName: "kubernetes.io/projected/566b2dae-dd3e-4c63-b4d2-e30667a5fd07-kube-api-access-cqn4t") pod "network-check-target-2qpkq" (UID: "566b2dae-dd3e-4c63-b4d2-e30667a5fd07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:26.918762 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:26.918726 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qpkq" Nov 02 22:47:26.918947 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:26.918874 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qpkq" podUID="566b2dae-dd3e-4c63-b4d2-e30667a5fd07" Nov 02 22:47:26.919020 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:26.918957 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:47:26.919076 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:26.919059 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-465bf" podUID="82b3b5e2-09f1-49c0-8d72-c85187918223" Nov 02 22:47:26.919139 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:26.919109 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:47:26.919238 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:26.919216 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbxxp" podUID="417d26e1-2a9d-41e3-8940-69626662a2f3" Nov 02 22:47:27.979006 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:27.978741 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.us-west-1.compute.internal" event={"ID":"38d433b21a6573d8bddad58877b72617","Type":"ContainerStarted","Data":"d670b52d4d4290795d2163b7672306d251c56590fd22e98d773214ad90aee696"} Nov 02 22:47:28.098508 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:28.097723 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.us-west-1.compute.internal" podStartSLOduration=6.097704644 podStartE2EDuration="6.097704644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:28.096829931 +0000 UTC m=+6.659193270" watchObservedRunningTime="2026-11-02 22:47:28.097704644 +0000 UTC m=+6.660067982" Nov 02 22:47:28.918321 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:28.918243 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:47:28.918488 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:28.918403 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbxxp" podUID="417d26e1-2a9d-41e3-8940-69626662a2f3" Nov 02 22:47:28.918557 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:28.918490 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qpkq" Nov 02 22:47:28.918611 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:28.918576 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qpkq" podUID="566b2dae-dd3e-4c63-b4d2-e30667a5fd07" Nov 02 22:47:28.918716 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:28.918691 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:47:28.918818 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:28.918800 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-465bf" podUID="82b3b5e2-09f1-49c0-8d72-c85187918223" Nov 02 22:47:28.984454 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:28.984413 2573 generic.go:356] "Generic (PLEG): container finished" podID="049a11ea66e964e484c1cdf0430a7a18" containerID="71fc147a02e78f80153778c3094bee5085c45b4464333ee554fc6d780b478554" exitCode=0 Nov 02 22:47:28.984945 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:28.984502 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal" event={"ID":"049a11ea66e964e484c1cdf0430a7a18","Type":"ContainerDied","Data":"71fc147a02e78f80153778c3094bee5085c45b4464333ee554fc6d780b478554"} Nov 02 22:47:28.990739 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:28.990642 2573 generic.go:356] "Generic (PLEG): container finished" podID="457675c6-c725-44e5-a857-97aa4c17e655" containerID="6cbea0ce2e4346e5fadef946dbea33fc4abb4b94e29de0dfc0411dc3199a0a65" exitCode=0 Nov 02 22:47:28.991597 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:28.991570 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9rnl" event={"ID":"457675c6-c725-44e5-a857-97aa4c17e655","Type":"ContainerDied","Data":"6cbea0ce2e4346e5fadef946dbea33fc4abb4b94e29de0dfc0411dc3199a0a65"} Nov 02 22:47:29.997866 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:29.997832 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal" event={"ID":"049a11ea66e964e484c1cdf0430a7a18","Type":"ContainerStarted","Data":"f15dc2bdc14df045bd1ac6f37ddb5a899d3f9c6658de31febddcfee628bfe087"} Nov 02 22:47:30.001615 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:30.001581 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r5lnw" event={"ID":"0ea8d3ef-20e9-4ecb-9e50-2df15d7da726","Type":"ContainerStarted","Data":"f74098e7a8ec6d9504f543cf29aa85c149eb630252fb08747dd0494766915258"} Nov 02 22:47:30.230930 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:30.230868 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal" podStartSLOduration=8.230847555 podStartE2EDuration="8.230847555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:30.113392393 +0000 UTC m=+8.675755733" watchObservedRunningTime="2026-11-02 22:47:30.230847555 +0000 UTC m=+8.793210893" Nov 02 22:47:30.771980 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:30.771947 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82b3b5e2-09f1-49c0-8d72-c85187918223-metrics-certs\") pod \"network-metrics-daemon-465bf\" (UID: \"82b3b5e2-09f1-49c0-8d72-c85187918223\") " pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:47:30.772180 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:30.771991 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/417d26e1-2a9d-41e3-8940-69626662a2f3-original-pull-secret\") pod \"global-pull-secret-syncer-wbxxp\" (UID: \"417d26e1-2a9d-41e3-8940-69626662a2f3\") " pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:47:30.772180 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:30.772111 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:30.772180 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:30.772175 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/417d26e1-2a9d-41e3-8940-69626662a2f3-original-pull-secret podName:417d26e1-2a9d-41e3-8940-69626662a2f3 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:38.772163407 +0000 UTC m=+17.334526733 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/417d26e1-2a9d-41e3-8940-69626662a2f3-original-pull-secret") pod "global-pull-secret-syncer-wbxxp" (UID: "417d26e1-2a9d-41e3-8940-69626662a2f3") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:30.772362 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:30.772219 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:30.772362 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:30.772245 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82b3b5e2-09f1-49c0-8d72-c85187918223-metrics-certs podName:82b3b5e2-09f1-49c0-8d72-c85187918223 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:38.772234902 +0000 UTC m=+17.334598232 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82b3b5e2-09f1-49c0-8d72-c85187918223-metrics-certs") pod "network-metrics-daemon-465bf" (UID: "82b3b5e2-09f1-49c0-8d72-c85187918223") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:30.872706 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:30.872613 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqn4t\" (UniqueName: \"kubernetes.io/projected/566b2dae-dd3e-4c63-b4d2-e30667a5fd07-kube-api-access-cqn4t\") pod \"network-check-target-2qpkq\" (UID: \"566b2dae-dd3e-4c63-b4d2-e30667a5fd07\") " pod="openshift-network-diagnostics/network-check-target-2qpkq" Nov 02 22:47:30.872908 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:30.872791 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:47:30.872908 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:30.872812 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:47:30.872908 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:30.872823 2573 projected.go:196] Error preparing data for projected volume kube-api-access-cqn4t for pod openshift-network-diagnostics/network-check-target-2qpkq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:30.872908 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:30.872889 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/566b2dae-dd3e-4c63-b4d2-e30667a5fd07-kube-api-access-cqn4t podName:566b2dae-dd3e-4c63-b4d2-e30667a5fd07 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:38.872871568 +0000 UTC m=+17.435234886 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cqn4t" (UniqueName: "kubernetes.io/projected/566b2dae-dd3e-4c63-b4d2-e30667a5fd07-kube-api-access-cqn4t") pod "network-check-target-2qpkq" (UID: "566b2dae-dd3e-4c63-b4d2-e30667a5fd07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:30.918862 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:30.918832 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:47:30.919032 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:30.918897 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qpkq" Nov 02 22:47:30.919032 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:30.918909 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:47:30.919032 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:30.918990 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qpkq" podUID="566b2dae-dd3e-4c63-b4d2-e30667a5fd07" Nov 02 22:47:30.919234 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:30.919066 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-465bf" podUID="82b3b5e2-09f1-49c0-8d72-c85187918223" Nov 02 22:47:30.919234 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:30.919157 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbxxp" podUID="417d26e1-2a9d-41e3-8940-69626662a2f3" Nov 02 22:47:31.007650 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:31.007618 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mp88w" event={"ID":"fc560bbe-b31a-48e8-8416-dda72f49dae1","Type":"ContainerStarted","Data":"b91620e569a1b2d49f8fbdbf7ec714067fdc78e8f38e74b5ab4cd154f4a3f6dd"} Nov 02 22:47:31.121809 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:31.121748 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-r5lnw" podStartSLOduration=4.163257759 podStartE2EDuration="9.121726591s" totalImagesPullingTime="4.958468832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:30.232175405 +0000 UTC m=+8.794538743" watchObservedRunningTime="2026-11-02 22:47:31.121726591 +0000 UTC m=+9.684089929" Nov 02 22:47:32.013572 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:32.013535 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" event={"ID":"fb045f55-e48d-4cac-88cf-90f90c225cbb","Type":"ContainerStarted","Data":"2885e052b259adadc5657fdfe557a40112ed1094f70bb7ebeda60ace32a62390"} Nov 02 22:47:32.919112 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:32.918373 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:47:32.919112 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:32.918639 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-465bf" podUID="82b3b5e2-09f1-49c0-8d72-c85187918223" Nov 02 22:47:32.919112 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:32.918749 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:47:32.919112 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:32.918873 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbxxp" podUID="417d26e1-2a9d-41e3-8940-69626662a2f3" Nov 02 22:47:32.919112 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:32.918966 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qpkq" Nov 02 22:47:32.919112 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:32.919059 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qpkq" podUID="566b2dae-dd3e-4c63-b4d2-e30667a5fd07" Nov 02 22:47:33.018458 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:33.018416 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mcvpd" event={"ID":"26b7328d-dacf-44c2-995b-306e3ef9cf31","Type":"ContainerStarted","Data":"ea1ccb70afdc7109e9ae7405518be43363ba6f75475524227a8068ada44b0f4f"} Nov 02 22:47:33.135804 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:33.135750 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mp88w" podStartSLOduration=5.2947161529999995 podStartE2EDuration="11.135736054s" totalImagesPullingTime="5.841019901s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:31.122180563 +0000 UTC m=+9.684543898" watchObservedRunningTime="2026-11-02 22:47:33.135736054 +0000 UTC m=+11.698099391" Nov 02 22:47:33.177703 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:33.177665 2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 22:47:33.852388 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:33.852007 2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T22:47:33.177690939Z","UUID":"462a851e-4d86-464a-a8da-991d6899e491","Handler":null,"Name":"","Endpoint":""} Nov 02 22:47:33.853694 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:33.853670 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 22:47:33.853694 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:33.853699 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 22:47:33.876218 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:33.876180 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-r5lnw" Nov 02 22:47:33.876844 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:33.876824 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-r5lnw" Nov 02 22:47:33.992879 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:33.992773 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-mcvpd" podStartSLOduration=3.554070904 podStartE2EDuration="11.992752759s" totalImagesPullingTime="8.438681855s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:33.136696177 +0000 UTC m=+11.699059515" watchObservedRunningTime="2026-11-02 22:47:33.992752759 +0000 UTC m=+12.555116097" Nov 02 22:47:34.022955 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:34.022914 2573 generic.go:356] "Generic (PLEG): container finished" podID="457675c6-c725-44e5-a857-97aa4c17e655" containerID="f550b5072b6e927ff4a84ccd13edcbc9ff98fa0aa1e9d415608123201a95bc3c" exitCode=0 Nov 02 22:47:34.023444 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:34.023006 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9rnl" event={"ID":"457675c6-c725-44e5-a857-97aa4c17e655","Type":"ContainerDied","Data":"f550b5072b6e927ff4a84ccd13edcbc9ff98fa0aa1e9d415608123201a95bc3c"} Nov 02 22:47:34.025802 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:34.025778 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" event={"ID":"fb045f55-e48d-4cac-88cf-90f90c225cbb","Type":"ContainerStarted","Data":"1583bacdc35e31cd9e8064895c1b9e263bb7db818dc8f2712020c773cf43420d"} Nov 02 22:47:34.032827 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:34.032796 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lhbwd" event={"ID":"cb204d6b-3758-4db4-bd68-0eacec87b805","Type":"ContainerStarted","Data":"289c267bc75fd1f189469f1ff2d9f5d8ebd391a652e1397e210d232d5a13853e"} Nov 02 22:47:34.258170 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:34.258046 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-lhbwd" podStartSLOduration=3.482196386 podStartE2EDuration="12.258029064s" totalImagesPullingTime="8.775832678s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:34.257916246 +0000 UTC m=+12.820279584" watchObservedRunningTime="2026-11-02 22:47:34.258029064 +0000 UTC m=+12.820392403" Nov 02 22:47:34.918650 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:34.918603 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:47:34.918836 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:34.918727 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbxxp" podUID="417d26e1-2a9d-41e3-8940-69626662a2f3" Nov 02 22:47:34.919139 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:34.919099 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qpkq" Nov 02 22:47:34.919251 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:34.919212 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qpkq" podUID="566b2dae-dd3e-4c63-b4d2-e30667a5fd07" Nov 02 22:47:34.919313 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:34.919278 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:47:34.919374 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:34.919357 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-465bf" podUID="82b3b5e2-09f1-49c0-8d72-c85187918223" Nov 02 22:47:35.036193 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:35.036121 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2kt76" event={"ID":"13a7b3bb-060c-4383-a864-0575834b221e","Type":"ContainerStarted","Data":"0a4c019b244c0ef41b73053ff034f6120b26ad3c834c7ef14067e20871d335d8"} Nov 02 22:47:35.038425 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:35.038394 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" event={"ID":"fb045f55-e48d-4cac-88cf-90f90c225cbb","Type":"ContainerStarted","Data":"8178ecc5fadf80265295d496c85dbfc968b0cc6f7855f0c98122cc11784984f2"} Nov 02 22:47:35.039969 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:35.039928 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rw8v4" event={"ID":"472c7524-6a70-420c-aa56-7f3028a53d52","Type":"ContainerStarted","Data":"7afb0f4ee05153a1a434267854aeabd531b7bdd1c6b2b40c05f261feae3e6a4c"} Nov 02 22:47:35.152658 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:35.152600 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2kt76" podStartSLOduration=2.623143259 podStartE2EDuration="13.152583399s" totalImagesPullingTime="10.52944014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:35.152042613 +0000 UTC m=+13.714405951" watchObservedRunningTime="2026-11-02 22:47:35.152583399 +0000 UTC m=+13.714946736" Nov 02 22:47:35.265279 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:35.265175 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rw8v4" podStartSLOduration=4.386843951 podStartE2EDuration="13.265165825s" totalImagesPullingTime="8.878321874s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:35.264563589 +0000 UTC m=+13.826926926" watchObservedRunningTime="2026-11-02 22:47:35.265165825 +0000 UTC m=+13.827529162" Nov 02 22:47:35.380793 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:35.380743 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" podStartSLOduration=3.469049942 podStartE2EDuration="13.380729046s" totalImagesPullingTime="9.911679104s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:47:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:35.380485607 +0000 UTC m=+13.942848945" watchObservedRunningTime="2026-11-02 22:47:35.380729046 +0000 UTC m=+13.943092382" Nov 02 22:47:36.043356 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:36.043324 2573 generic.go:356] "Generic (PLEG): container finished" podID="457675c6-c725-44e5-a857-97aa4c17e655" containerID="eeb49842f57448e9799fdf4c141cba32728d039ca116aee8fe132ea734e04257" exitCode=0 Nov 02 22:47:36.043877 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:36.043427 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9rnl" event={"ID":"457675c6-c725-44e5-a857-97aa4c17e655","Type":"ContainerDied","Data":"eeb49842f57448e9799fdf4c141cba32728d039ca116aee8fe132ea734e04257"} Nov 02 22:47:36.918642 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:36.918597 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qpkq" Nov 02 22:47:36.918804 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:36.918597 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:47:36.918804 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:36.918725 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qpkq" podUID="566b2dae-dd3e-4c63-b4d2-e30667a5fd07" Nov 02 22:47:36.918804 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:36.918613 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:47:36.918804 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:36.918784 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-465bf" podUID="82b3b5e2-09f1-49c0-8d72-c85187918223" Nov 02 22:47:36.919014 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:36.918911 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbxxp" podUID="417d26e1-2a9d-41e3-8940-69626662a2f3" Nov 02 22:47:38.050831 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:38.050789 2573 generic.go:356] "Generic (PLEG): container finished" podID="457675c6-c725-44e5-a857-97aa4c17e655" containerID="249507aa03b4c41f387caa1326fcb1cd1b71ccef44b5321f9158deab71887385" exitCode=0 Nov 02 22:47:38.051547 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:38.050860 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9rnl" event={"ID":"457675c6-c725-44e5-a857-97aa4c17e655","Type":"ContainerDied","Data":"249507aa03b4c41f387caa1326fcb1cd1b71ccef44b5321f9158deab71887385"} Nov 02 22:47:38.847569 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:38.847354 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82b3b5e2-09f1-49c0-8d72-c85187918223-metrics-certs\") pod \"network-metrics-daemon-465bf\" (UID: \"82b3b5e2-09f1-49c0-8d72-c85187918223\") " pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:47:38.847755 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:38.847587 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/417d26e1-2a9d-41e3-8940-69626662a2f3-original-pull-secret\") pod \"global-pull-secret-syncer-wbxxp\" (UID: \"417d26e1-2a9d-41e3-8940-69626662a2f3\") " pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:47:38.847755 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:38.847525 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:38.847755 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:38.847713 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82b3b5e2-09f1-49c0-8d72-c85187918223-metrics-certs podName:82b3b5e2-09f1-49c0-8d72-c85187918223 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:54.847694205 +0000 UTC m=+33.410057544 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82b3b5e2-09f1-49c0-8d72-c85187918223-metrics-certs") pod "network-metrics-daemon-465bf" (UID: "82b3b5e2-09f1-49c0-8d72-c85187918223") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:38.847755 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:38.847729 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:38.847941 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:38.847774 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/417d26e1-2a9d-41e3-8940-69626662a2f3-original-pull-secret podName:417d26e1-2a9d-41e3-8940-69626662a2f3 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:54.847763302 +0000 UTC m=+33.410126617 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/417d26e1-2a9d-41e3-8940-69626662a2f3-original-pull-secret") pod "global-pull-secret-syncer-wbxxp" (UID: "417d26e1-2a9d-41e3-8940-69626662a2f3") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:38.918511 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:38.918478 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:47:38.918682 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:38.918543 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:47:38.918682 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:38.918571 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qpkq" Nov 02 22:47:38.918682 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:38.918655 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-465bf" podUID="82b3b5e2-09f1-49c0-8d72-c85187918223" Nov 02 22:47:38.918814 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:38.918709 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qpkq" podUID="566b2dae-dd3e-4c63-b4d2-e30667a5fd07" Nov 02 22:47:38.918814 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:38.918733 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbxxp" podUID="417d26e1-2a9d-41e3-8940-69626662a2f3" Nov 02 22:47:38.948344 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:38.948306 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqn4t\" (UniqueName: \"kubernetes.io/projected/566b2dae-dd3e-4c63-b4d2-e30667a5fd07-kube-api-access-cqn4t\") pod \"network-check-target-2qpkq\" (UID: \"566b2dae-dd3e-4c63-b4d2-e30667a5fd07\") " pod="openshift-network-diagnostics/network-check-target-2qpkq" Nov 02 22:47:38.948501 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:38.948484 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:47:38.948557 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:38.948503 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:47:38.948557 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:38.948514 2573 projected.go:196] Error preparing data for projected volume kube-api-access-cqn4t for pod openshift-network-diagnostics/network-check-target-2qpkq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:38.948654 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:38.948568 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/566b2dae-dd3e-4c63-b4d2-e30667a5fd07-kube-api-access-cqn4t podName:566b2dae-dd3e-4c63-b4d2-e30667a5fd07 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:54.948556032 +0000 UTC m=+33.510919351 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cqn4t" (UniqueName: "kubernetes.io/projected/566b2dae-dd3e-4c63-b4d2-e30667a5fd07-kube-api-access-cqn4t") pod "network-check-target-2qpkq" (UID: "566b2dae-dd3e-4c63-b4d2-e30667a5fd07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:40.918238 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:40.917982 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:47:40.918578 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:40.917979 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:47:40.918578 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:40.918273 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbxxp" podUID="417d26e1-2a9d-41e3-8940-69626662a2f3" Nov 02 22:47:40.918578 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:40.918030 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qpkq" Nov 02 22:47:40.918578 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:40.918345 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-465bf" podUID="82b3b5e2-09f1-49c0-8d72-c85187918223" Nov 02 22:47:40.918578 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:40.918398 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qpkq" podUID="566b2dae-dd3e-4c63-b4d2-e30667a5fd07" Nov 02 22:47:41.060919 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:41.060884 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" event={"ID":"7321845c-9a95-415b-b6db-2bf983d77fa5","Type":"ContainerStarted","Data":"da7f9849f9c34a27484ebd9a6b94f33d85f18f76d727baec3fcc03cdb4ebc6cd"} Nov 02 22:47:41.061107 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:41.060971 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" event={"ID":"7321845c-9a95-415b-b6db-2bf983d77fa5","Type":"ContainerStarted","Data":"8f2fdc065bb40938ed75c39cc34aa68239c6457eda3e2e105b9bacf9d6cd8ffa"} Nov 02 22:47:41.061107 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:41.061014 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" event={"ID":"7321845c-9a95-415b-b6db-2bf983d77fa5","Type":"ContainerStarted","Data":"f43fd63fbcce5a32a3dafeffa2ebfdbaf647d456f4964e26a23fc1e3af038bee"} Nov 02 22:47:41.061107 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:41.061054 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" event={"ID":"7321845c-9a95-415b-b6db-2bf983d77fa5","Type":"ContainerStarted","Data":"8ac4d9f88bb2b46f69d45c3771588a031248e578df2d79d811e39cc0071cfca8"} Nov 02 22:47:41.061107 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:41.061094 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" event={"ID":"7321845c-9a95-415b-b6db-2bf983d77fa5","Type":"ContainerStarted","Data":"ffb93f777291fe7cafc9181bfef4dc126e7d712f3e1656e9dfb9bcf5326c9895"} Nov 02 22:47:41.061301 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:41.061156 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" event={"ID":"7321845c-9a95-415b-b6db-2bf983d77fa5","Type":"ContainerStarted","Data":"5acc6964ddfd10c9a3025296feef5d01341d8a51e6e3196a04e1bc3955d32800"} Nov 02 22:47:41.063768 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:41.063742 2573 generic.go:356] "Generic (PLEG): container finished" podID="457675c6-c725-44e5-a857-97aa4c17e655" containerID="2a408bee88ce403cbf74d4ef36e72f27962ed45de0c5b14df68f49cd1cc34728" exitCode=0 Nov 02 22:47:41.063875 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:41.063781 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9rnl" event={"ID":"457675c6-c725-44e5-a857-97aa4c17e655","Type":"ContainerDied","Data":"2a408bee88ce403cbf74d4ef36e72f27962ed45de0c5b14df68f49cd1cc34728"} Nov 02 22:47:42.069079 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:42.069045 2573 generic.go:356] "Generic (PLEG): container finished" podID="457675c6-c725-44e5-a857-97aa4c17e655" containerID="0d91fe2ba2bbe3d4b3f55c4ee701f5fb27f51bd238da250d54d46d81d4505723" exitCode=0 Nov 02 22:47:42.069547 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:42.069106 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9rnl" event={"ID":"457675c6-c725-44e5-a857-97aa4c17e655","Type":"ContainerDied","Data":"0d91fe2ba2bbe3d4b3f55c4ee701f5fb27f51bd238da250d54d46d81d4505723"} Nov 02 22:47:42.917907 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:42.917863 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:47:42.917907 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:42.917889 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qpkq" Nov 02 22:47:42.918162 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:42.917884 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:47:42.918162 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:42.917978 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-465bf" podUID="82b3b5e2-09f1-49c0-8d72-c85187918223" Nov 02 22:47:42.918162 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:42.918065 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbxxp" podUID="417d26e1-2a9d-41e3-8940-69626662a2f3" Nov 02 22:47:42.918281 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:42.918174 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qpkq" podUID="566b2dae-dd3e-4c63-b4d2-e30667a5fd07" Nov 02 22:47:43.075021 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:43.074988 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9rnl" event={"ID":"457675c6-c725-44e5-a857-97aa4c17e655","Type":"ContainerStarted","Data":"904f8b99dcf0aa9edfa81a25f08fa266299d2a72ad5e232c48af241de0a4528a"} Nov 02 22:47:43.078241 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:43.078210 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" event={"ID":"7321845c-9a95-415b-b6db-2bf983d77fa5","Type":"ContainerStarted","Data":"64752c5ee1ecc9132beb366b68699060f33b21bec6344be111b91462d9228707"} Nov 02 22:47:43.201669 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:43.201615 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-t9rnl" podStartSLOduration=8.296230564 podStartE2EDuration="21.201603056s" totalImagesPullingTime="12.905372492s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:47:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:43.201087724 +0000 UTC m=+21.763451061" watchObservedRunningTime="2026-11-02 22:47:43.201603056 +0000 UTC m=+21.763966393" Nov 02 22:47:44.918269 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:44.918233 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:47:44.918269 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:44.918254 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qpkq" Nov 02 22:47:44.918871 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:44.918366 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-465bf" podUID="82b3b5e2-09f1-49c0-8d72-c85187918223" Nov 02 22:47:44.918871 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:44.918455 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qpkq" podUID="566b2dae-dd3e-4c63-b4d2-e30667a5fd07" Nov 02 22:47:44.918871 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:44.918509 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:47:44.918871 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:44.918617 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbxxp" podUID="417d26e1-2a9d-41e3-8940-69626662a2f3" Nov 02 22:47:45.086558 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:45.086521 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" event={"ID":"7321845c-9a95-415b-b6db-2bf983d77fa5","Type":"ContainerStarted","Data":"487cdf72ea34b2ce7e6c493dc817c6f835bc364acf73ddfb2c9b7b9447baebfe"} Nov 02 22:47:45.087103 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:45.087076 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:45.087295 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:45.087213 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:45.087295 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:45.087273 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:45.104082 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:45.104021 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:45.104756 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:45.104741 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:47:45.361026 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:45.360920 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" podStartSLOduration=7.466546819 podStartE2EDuration="23.360907576s" totalImagesPullingTime="15.894360757s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:45.311377831 +0000 UTC m=+23.873741167" watchObservedRunningTime="2026-11-02 22:47:45.360907576 +0000 UTC m=+23.923270913" Nov 02 22:47:46.918837 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:46.918657 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:47:46.919278 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:46.918683 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:47:46.919278 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:46.918908 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbxxp" podUID="417d26e1-2a9d-41e3-8940-69626662a2f3" Nov 02 22:47:46.919278 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:46.918727 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qpkq" Nov 02 22:47:46.919278 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:46.918967 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-465bf" podUID="82b3b5e2-09f1-49c0-8d72-c85187918223" Nov 02 22:47:46.919278 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:46.919041 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qpkq" podUID="566b2dae-dd3e-4c63-b4d2-e30667a5fd07" Nov 02 22:47:47.610484 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:47.610450 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wbxxp"] Nov 02 22:47:47.610652 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:47.610552 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:47:47.610652 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:47.610633 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbxxp" podUID="417d26e1-2a9d-41e3-8940-69626662a2f3" Nov 02 22:47:47.613509 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:47.613481 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2qpkq"] Nov 02 22:47:47.613646 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:47.613588 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qpkq" Nov 02 22:47:47.613715 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:47.613658 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qpkq" podUID="566b2dae-dd3e-4c63-b4d2-e30667a5fd07" Nov 02 22:47:47.614298 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:47.614261 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-465bf"] Nov 02 22:47:47.614402 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:47.614354 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:47:47.614448 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:47.614434 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-465bf" podUID="82b3b5e2-09f1-49c0-8d72-c85187918223" Nov 02 22:47:48.917943 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:48.917904 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:47:48.917943 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:48.917938 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:47:48.918505 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:48.918057 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-465bf" podUID="82b3b5e2-09f1-49c0-8d72-c85187918223" Nov 02 22:47:48.918505 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:48.918180 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbxxp" podUID="417d26e1-2a9d-41e3-8940-69626662a2f3" Nov 02 22:47:49.918720 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:49.918689 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qpkq" Nov 02 22:47:49.919094 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:49.918802 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qpkq" podUID="566b2dae-dd3e-4c63-b4d2-e30667a5fd07" Nov 02 22:47:50.917895 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:50.917861 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:47:50.918043 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:50.917968 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbxxp" podUID="417d26e1-2a9d-41e3-8940-69626662a2f3" Nov 02 22:47:50.918043 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:50.918016 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:47:50.918157 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:50.918112 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-465bf" podUID="82b3b5e2-09f1-49c0-8d72-c85187918223" Nov 02 22:47:51.919116 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:51.919089 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qpkq" Nov 02 22:47:51.919499 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:51.919200 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qpkq" podUID="566b2dae-dd3e-4c63-b4d2-e30667a5fd07" Nov 02 22:47:52.215239 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:52.215106 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-r5lnw" Nov 02 22:47:52.215402 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:52.215353 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 22:47:52.215746 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:52.215717 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-r5lnw" Nov 02 22:47:52.918559 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:52.918526 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:47:52.918748 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:52.918526 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:47:52.918748 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:52.918624 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbxxp" podUID="417d26e1-2a9d-41e3-8940-69626662a2f3" Nov 02 22:47:52.918748 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:52.918731 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-465bf" podUID="82b3b5e2-09f1-49c0-8d72-c85187918223" Nov 02 22:47:53.734747 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:53.734710 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-176.us-west-1.compute.internal" event="NodeReady" Nov 02 22:47:53.735104 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:53.734845 2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 22:47:53.848753 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:53.848568 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5wnxz"] Nov 02 22:47:53.869586 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:53.869563 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5wnxz" Nov 02 22:47:53.883001 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:53.882922 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-25msk"] Nov 02 22:47:53.893504 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:53.893482 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-276wh"] Nov 02 22:47:53.893623 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:53.893607 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-25msk" Nov 02 22:47:53.914602 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:53.914574 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-qcz4m"] Nov 02 22:47:53.914752 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:53.914674 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-276wh" Nov 02 22:47:53.931067 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:53.931035 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt"] Nov 02 22:47:53.931067 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:53.931058 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qcz4m" Nov 02 22:47:53.931313 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:53.931187 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qpkq" Nov 02 22:47:53.937217 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:53.937195 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 02 22:47:53.937503 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:53.937479 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 02 22:47:53.957653 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:53.955830 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnf2q\" (UniqueName: \"kubernetes.io/projected/a8d56c63-207c-42b3-8946-c993d0fde66b-kube-api-access-tnf2q\") pod \"kas-connection-checker-685895c7c9-5wnxz\" (UID: \"a8d56c63-207c-42b3-8946-c993d0fde66b\") " pod="kube-system/kas-connection-checker-685895c7c9-5wnxz" Nov 02 22:47:53.959988 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:53.959503 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt" Nov 02 22:47:53.964752 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:53.964726 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-fj4n7"] Nov 02 22:47:53.988982 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:53.988958 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 02 22:47:53.989089 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:53.989049 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wk6k8\"" Nov 02 22:47:53.989210 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:53.989194 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-464jz"] Nov 02 22:47:53.989328 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:53.989313 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" Nov 02 22:47:54.011265 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.011238 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-566bb4455-kq2j4"] Nov 02 22:47:54.012732 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.011320 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-464jz" Nov 02 22:47:54.021595 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.021569 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 22:47:54.021858 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.021841 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 22:47:54.022969 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.022935 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 02 22:47:54.023094 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.023029 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 22:47:54.023181 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.023100 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 22:47:54.023181 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.023161 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:47:54.041358 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.041335 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5ljqt"] Nov 02 22:47:54.041473 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.041456 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:47:54.056092 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.056069 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsmhx\" (UniqueName: \"kubernetes.io/projected/12ce41df-43ef-4b00-be8e-5d3fad591eea-kube-api-access-gsmhx\") pod \"kas-connection-checker-685895c7c9-276wh\" (UID: \"12ce41df-43ef-4b00-be8e-5d3fad591eea\") " pod="kube-system/kas-connection-checker-685895c7c9-276wh" Nov 02 22:47:54.056204 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.056113 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnf2q\" (UniqueName: \"kubernetes.io/projected/a8d56c63-207c-42b3-8946-c993d0fde66b-kube-api-access-tnf2q\") pod \"kas-connection-checker-685895c7c9-5wnxz\" (UID: \"a8d56c63-207c-42b3-8946-c993d0fde66b\") " pod="kube-system/kas-connection-checker-685895c7c9-5wnxz" Nov 02 22:47:54.056253 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.056211 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbvvt\" (UniqueName: \"kubernetes.io/projected/53ba063f-0db3-4f82-b280-40718c28b479-kube-api-access-jbvvt\") pod \"cluster-monitoring-operator-884576c57-qcz4m\" (UID: \"53ba063f-0db3-4f82-b280-40718c28b479\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qcz4m" Nov 02 22:47:54.056253 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.056239 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/53ba063f-0db3-4f82-b280-40718c28b479-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-qcz4m\" (UID: \"53ba063f-0db3-4f82-b280-40718c28b479\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qcz4m" Nov 02 22:47:54.056314 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.056299 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae96a9c1-10a6-4050-bca0-6a2e7859d3c3-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt\" (UID: \"ae96a9c1-10a6-4050-bca0-6a2e7859d3c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt" Nov 02 22:47:54.056342 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.056334 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjvmh\" (UniqueName: \"kubernetes.io/projected/ecdc26b5-be99-4928-8fff-203cc2c19ad6-kube-api-access-jjvmh\") pod \"kas-connection-checker-685895c7c9-25msk\" (UID: \"ecdc26b5-be99-4928-8fff-203cc2c19ad6\") " pod="kube-system/kas-connection-checker-685895c7c9-25msk" Nov 02 22:47:54.056375 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.056357 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qw9l\" (UniqueName: \"kubernetes.io/projected/ae96a9c1-10a6-4050-bca0-6a2e7859d3c3-kube-api-access-9qw9l\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt\" (UID: \"ae96a9c1-10a6-4050-bca0-6a2e7859d3c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt" Nov 02 22:47:54.056416 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.056382 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ba063f-0db3-4f82-b280-40718c28b479-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qcz4m\" (UID: \"53ba063f-0db3-4f82-b280-40718c28b479\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qcz4m" Nov 02 22:47:54.056503 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.056483 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae96a9c1-10a6-4050-bca0-6a2e7859d3c3-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt\" (UID: \"ae96a9c1-10a6-4050-bca0-6a2e7859d3c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt" Nov 02 22:47:54.065634 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.065614 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-xglvd"] Nov 02 22:47:54.065755 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.065739 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5ljqt" Nov 02 22:47:54.081657 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.081631 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rf5x6"] Nov 02 22:47:54.081765 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.081747 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xglvd" Nov 02 22:47:54.093402 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.093374 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 02 22:47:54.093402 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.093405 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 02 22:47:54.093564 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.093442 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 02 22:47:54.093564 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.093457 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 02 22:47:54.093564 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.093516 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 02 22:47:54.093564 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.093539 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 02 22:47:54.094577 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.094563 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-86798bc874-lbrfv"] Nov 02 22:47:54.094680 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.094666 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rf5x6" Nov 02 22:47:54.112684 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.112659 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-nj7zh"] Nov 02 22:47:54.112810 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.112793 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:47:54.119742 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.119721 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnf2q\" (UniqueName: \"kubernetes.io/projected/a8d56c63-207c-42b3-8946-c993d0fde66b-kube-api-access-tnf2q\") pod \"kas-connection-checker-685895c7c9-5wnxz\" (UID: \"a8d56c63-207c-42b3-8946-c993d0fde66b\") " pod="kube-system/kas-connection-checker-685895c7c9-5wnxz" Nov 02 22:47:54.123148 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.123111 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 22:47:54.123313 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.123296 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:47:54.123379 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.123340 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 02 22:47:54.123552 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.123535 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 02 22:47:54.123649 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.123625 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 02 22:47:54.123758 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.123660 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 22:47:54.123758 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.123672 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 22:47:54.123758 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.123686 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 02 22:47:54.123758 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.123707 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 02 22:47:54.123758 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.123717 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:47:54.123758 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.123750 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 22:47:54.124019 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.123781 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 22:47:54.124019 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.123880 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 02 22:47:54.132978 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.132959 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-nbswd"] Nov 02 22:47:54.133120 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.133109 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-nj7zh" Nov 02 22:47:54.146595 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.146580 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-bc8cbc8c-hghqr"] Nov 02 22:47:54.146710 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.146696 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nbswd" Nov 02 22:47:54.156936 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.156917 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjvmh\" (UniqueName: \"kubernetes.io/projected/ecdc26b5-be99-4928-8fff-203cc2c19ad6-kube-api-access-jjvmh\") pod \"kas-connection-checker-685895c7c9-25msk\" (UID: \"ecdc26b5-be99-4928-8fff-203cc2c19ad6\") " pod="kube-system/kas-connection-checker-685895c7c9-25msk" Nov 02 22:47:54.157030 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.156949 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qw9l\" (UniqueName: \"kubernetes.io/projected/ae96a9c1-10a6-4050-bca0-6a2e7859d3c3-kube-api-access-9qw9l\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt\" (UID: \"ae96a9c1-10a6-4050-bca0-6a2e7859d3c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt" Nov 02 22:47:54.157030 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.156980 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvp6g\" (UniqueName: \"kubernetes.io/projected/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-kube-api-access-zvp6g\") pod \"insights-operator-7c4986d449-464jz\" (UID: \"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c\") " pod="openshift-insights/insights-operator-7c4986d449-464jz" Nov 02 22:47:54.157030 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.157005 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5-config\") pod \"console-operator-7cd56b8dbc-fj4n7\" (UID: \"a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" Nov 02 22:47:54.157180 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.157029 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5bg9\" (UniqueName: \"kubernetes.io/projected/1dccc35b-6206-47f3-b901-bafc5f17b8ad-kube-api-access-s5bg9\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:47:54.157180 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.157058 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-serving-cert\") pod \"insights-operator-7c4986d449-464jz\" (UID: \"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c\") " pod="openshift-insights/insights-operator-7c4986d449-464jz" Nov 02 22:47:54.157180 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.157172 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-default-certificate\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:47:54.157269 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.157239 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ba063f-0db3-4f82-b280-40718c28b479-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qcz4m\" (UID: \"53ba063f-0db3-4f82-b280-40718c28b479\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qcz4m" Nov 02 22:47:54.157298 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.157286 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-metrics-certs\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:47:54.157354 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.157338 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 22:47:54.157413 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.157402 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ba063f-0db3-4f82-b280-40718c28b479-cluster-monitoring-operator-tls podName:53ba063f-0db3-4f82-b280-40718c28b479 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:54.657382536 +0000 UTC m=+33.219745851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/53ba063f-0db3-4f82-b280-40718c28b479-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qcz4m" (UID: "53ba063f-0db3-4f82-b280-40718c28b479") : secret "cluster-monitoring-operator-tls" not found Nov 02 22:47:54.157450 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.157422 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae96a9c1-10a6-4050-bca0-6a2e7859d3c3-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt\" (UID: \"ae96a9c1-10a6-4050-bca0-6a2e7859d3c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt" Nov 02 22:47:54.157450 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.157440 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-snapshots\") pod \"insights-operator-7c4986d449-464jz\" (UID: \"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c\") " pod="openshift-insights/insights-operator-7c4986d449-464jz" Nov 02 22:47:54.157514 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.157471 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-stats-auth\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:47:54.157615 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.157600 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsmhx\" (UniqueName: \"kubernetes.io/projected/12ce41df-43ef-4b00-be8e-5d3fad591eea-kube-api-access-gsmhx\") pod \"kas-connection-checker-685895c7c9-276wh\" (UID: \"12ce41df-43ef-4b00-be8e-5d3fad591eea\") " pod="kube-system/kas-connection-checker-685895c7c9-276wh" Nov 02 22:47:54.157672 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.157654 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5-trusted-ca\") pod \"console-operator-7cd56b8dbc-fj4n7\" (UID: \"a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" Nov 02 22:47:54.157793 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.157781 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbvvt\" (UniqueName: \"kubernetes.io/projected/53ba063f-0db3-4f82-b280-40718c28b479-kube-api-access-jbvvt\") pod \"cluster-monitoring-operator-884576c57-qcz4m\" (UID: \"53ba063f-0db3-4f82-b280-40718c28b479\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qcz4m" Nov 02 22:47:54.157842 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.157801 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-service-ca-bundle\") pod \"insights-operator-7c4986d449-464jz\" (UID: \"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c\") " pod="openshift-insights/insights-operator-7c4986d449-464jz" Nov 02 22:47:54.157842 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.157815 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dccc35b-6206-47f3-b901-bafc5f17b8ad-service-ca-bundle\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:47:54.157930 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.157854 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjlw7\" (UniqueName: \"kubernetes.io/projected/ca5741f4-603c-448e-9088-0543eef08544-kube-api-access-bjlw7\") pod \"volume-data-source-validator-58c9f5f46d-5ljqt\" (UID: \"ca5741f4-603c-448e-9088-0543eef08544\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5ljqt" Nov 02 22:47:54.157966 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.157942 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfz5j\" (UniqueName: \"kubernetes.io/projected/a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5-kube-api-access-bfz5j\") pod \"console-operator-7cd56b8dbc-fj4n7\" (UID: \"a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" Nov 02 22:47:54.158008 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.157988 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/53ba063f-0db3-4f82-b280-40718c28b479-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-qcz4m\" (UID: \"53ba063f-0db3-4f82-b280-40718c28b479\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qcz4m" Nov 02 22:47:54.158042 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.158033 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae96a9c1-10a6-4050-bca0-6a2e7859d3c3-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt\" (UID: \"ae96a9c1-10a6-4050-bca0-6a2e7859d3c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt" Nov 02 22:47:54.158184 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.158169 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-tmp\") pod \"insights-operator-7c4986d449-464jz\" (UID: \"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c\") " pod="openshift-insights/insights-operator-7c4986d449-464jz" Nov 02 22:47:54.158252 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.158191 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-464jz\" (UID: \"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c\") " pod="openshift-insights/insights-operator-7c4986d449-464jz" Nov 02 22:47:54.158252 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.158210 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5-serving-cert\") pod \"console-operator-7cd56b8dbc-fj4n7\" (UID: \"a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" Nov 02 22:47:54.158742 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.158715 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae96a9c1-10a6-4050-bca0-6a2e7859d3c3-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt\" (UID: \"ae96a9c1-10a6-4050-bca0-6a2e7859d3c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt" Nov 02 22:47:54.158800 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.158727 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/53ba063f-0db3-4f82-b280-40718c28b479-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-qcz4m\" (UID: \"53ba063f-0db3-4f82-b280-40718c28b479\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qcz4m" Nov 02 22:47:54.159864 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.159846 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae96a9c1-10a6-4050-bca0-6a2e7859d3c3-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt\" (UID: \"ae96a9c1-10a6-4050-bca0-6a2e7859d3c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt" Nov 02 22:47:54.164622 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.164600 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6jd4b"] Nov 02 22:47:54.164749 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.164735 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:47:54.173214 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.173194 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 02 22:47:54.173276 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.173221 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:47:54.173276 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.173227 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 02 22:47:54.173276 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.173267 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 22:47:54.173378 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.173306 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 02 22:47:54.182873 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.182848 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-c8kg9"] Nov 02 22:47:54.182986 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.182952 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6jd4b" Nov 02 22:47:54.189424 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.189407 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 02 22:47:54.189516 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.189439 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 22:47:54.189516 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.189465 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 02 22:47:54.189516 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.189494 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 22:47:54.189621 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.189535 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 02 22:47:54.189621 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.189586 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 22:47:54.189621 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.189587 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-rncv6\"" Nov 02 22:47:54.194627 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.194614 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5wnxz" Nov 02 22:47:54.205250 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.205230 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-276wh"] Nov 02 22:47:54.205314 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.205259 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5wnxz"] Nov 02 22:47:54.205314 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.205274 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-25msk"] Nov 02 22:47:54.205314 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.205288 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-qcz4m"] Nov 02 22:47:54.205314 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.205307 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt"] Nov 02 22:47:54.205492 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.205322 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-fj4n7"] Nov 02 22:47:54.205492 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.205335 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-464jz"] Nov 02 22:47:54.205492 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.205347 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-xglvd"] Nov 02 22:47:54.205492 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.205361 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-nj7zh"] Nov 02 22:47:54.205492 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.205367 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c8kg9" Nov 02 22:47:54.205492 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.205374 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rf5x6"] Nov 02 22:47:54.205492 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.205472 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bc8cbc8c-hghqr"] Nov 02 22:47:54.205492 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.205489 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-566bb4455-kq2j4"] Nov 02 22:47:54.205838 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.205503 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5ljqt"] Nov 02 22:47:54.205838 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.205515 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-nbswd"] Nov 02 22:47:54.205838 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.205527 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c8kg9"] Nov 02 22:47:54.205838 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.205575 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86798bc874-lbrfv"] Nov 02 22:47:54.205838 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.205623 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6jd4b"] Nov 02 22:47:54.218263 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.218245 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjvmh\" (UniqueName: \"kubernetes.io/projected/ecdc26b5-be99-4928-8fff-203cc2c19ad6-kube-api-access-jjvmh\") pod \"kas-connection-checker-685895c7c9-25msk\" (UID: \"ecdc26b5-be99-4928-8fff-203cc2c19ad6\") " pod="kube-system/kas-connection-checker-685895c7c9-25msk" Nov 02 22:47:54.218342 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.218323 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qw9l\" (UniqueName: \"kubernetes.io/projected/ae96a9c1-10a6-4050-bca0-6a2e7859d3c3-kube-api-access-9qw9l\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt\" (UID: \"ae96a9c1-10a6-4050-bca0-6a2e7859d3c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt" Nov 02 22:47:54.218548 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.218530 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbvvt\" (UniqueName: \"kubernetes.io/projected/53ba063f-0db3-4f82-b280-40718c28b479-kube-api-access-jbvvt\") pod \"cluster-monitoring-operator-884576c57-qcz4m\" (UID: \"53ba063f-0db3-4f82-b280-40718c28b479\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qcz4m" Nov 02 22:47:54.222651 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.222631 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsmhx\" (UniqueName: \"kubernetes.io/projected/12ce41df-43ef-4b00-be8e-5d3fad591eea-kube-api-access-gsmhx\") pod \"kas-connection-checker-685895c7c9-276wh\" (UID: \"12ce41df-43ef-4b00-be8e-5d3fad591eea\") " pod="kube-system/kas-connection-checker-685895c7c9-276wh" Nov 02 22:47:54.225261 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.224831 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:47:54.225261 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.224974 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 02 22:47:54.225261 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.225087 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-25msk" Nov 02 22:47:54.225578 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.225493 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 02 22:47:54.225578 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.225497 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 02 22:47:54.232684 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.232660 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 22:47:54.239834 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.239816 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-276wh" Nov 02 22:47:54.258604 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.258581 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-registry-tls\") pod \"image-registry-86798bc874-lbrfv\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:47:54.258702 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.258624 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-service-ca-bundle\") pod \"insights-operator-7c4986d449-464jz\" (UID: \"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c\") " pod="openshift-insights/insights-operator-7c4986d449-464jz" Nov 02 22:47:54.258702 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.258643 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjlw7\" (UniqueName: \"kubernetes.io/projected/ca5741f4-603c-448e-9088-0543eef08544-kube-api-access-bjlw7\") pod \"volume-data-source-validator-58c9f5f46d-5ljqt\" (UID: \"ca5741f4-603c-448e-9088-0543eef08544\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5ljqt" Nov 02 22:47:54.258802 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.258767 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfz5j\" (UniqueName: \"kubernetes.io/projected/a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5-kube-api-access-bfz5j\") pod \"console-operator-7cd56b8dbc-fj4n7\" (UID: \"a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" Nov 02 22:47:54.258859 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.258812 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-rf5x6\" (UID: \"2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rf5x6" Nov 02 22:47:54.258859 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.258850 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5-serving-cert\") pod \"console-operator-7cd56b8dbc-fj4n7\" (UID: \"a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" Nov 02 22:47:54.258963 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.258877 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-tls\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:47:54.258963 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.258902 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-trusted-ca\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:47:54.258963 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.258931 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-installation-pull-secrets\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:47:54.259092 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.259070 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5bg9\" (UniqueName: \"kubernetes.io/projected/1dccc35b-6206-47f3-b901-bafc5f17b8ad-kube-api-access-s5bg9\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:47:54.259196 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.259173 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/071262de-534b-4d01-bc20-784db9852f69-serving-cert\") pod \"service-ca-operator-544d749c8c-xglvd\" (UID: \"071262de-534b-4d01-bc20-784db9852f69\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xglvd" Nov 02 22:47:54.259284 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.259212 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6cmc\" (UniqueName: \"kubernetes.io/projected/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-kube-api-access-t6cmc\") pod \"cluster-samples-operator-6ddb6c8b94-rf5x6\" (UID: \"2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rf5x6" Nov 02 22:47:54.259380 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.259358 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgmgr\" (UniqueName: \"kubernetes.io/projected/ff5ad4d1-4fd9-4374-b11c-78596c9cafa8-kube-api-access-cgmgr\") pod \"network-check-source-6f78d59f99-nbswd\" (UID: \"ff5ad4d1-4fd9-4374-b11c-78596c9cafa8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nbswd" Nov 02 22:47:54.259380 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.259371 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-service-ca-bundle\") pod \"insights-operator-7c4986d449-464jz\" (UID: \"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c\") " pod="openshift-insights/insights-operator-7c4986d449-464jz" Nov 02 22:47:54.259551 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.259421 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-bound-sa-token\") pod \"image-registry-86798bc874-lbrfv\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:47:54.259551 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.259522 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-bound-sa-token\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:47:54.259635 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.259600 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-snapshots\") pod \"insights-operator-7c4986d449-464jz\" (UID: \"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c\") " pod="openshift-insights/insights-operator-7c4986d449-464jz" Nov 02 22:47:54.259635 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.259630 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-stats-auth\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:47:54.259905 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.259845 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/db94cbdb-79e6-4d1d-bf66-269fbab94abe-image-registry-private-configuration\") pod \"image-registry-86798bc874-lbrfv\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:47:54.259995 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.259906 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/db94cbdb-79e6-4d1d-bf66-269fbab94abe-registry-certificates\") pod \"image-registry-86798bc874-lbrfv\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:47:54.260048 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.259998 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/db94cbdb-79e6-4d1d-bf66-269fbab94abe-installation-pull-secrets\") pod \"image-registry-86798bc874-lbrfv\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:47:54.260095 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.260068 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5-trusted-ca\") pod \"console-operator-7cd56b8dbc-fj4n7\" (UID: \"a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" Nov 02 22:47:54.260145 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.260101 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542-nginx-conf\") pod \"networking-console-plugin-86c478bb88-nj7zh\" (UID: \"d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nj7zh" Nov 02 22:47:54.260182 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.260100 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-snapshots\") pod \"insights-operator-7c4986d449-464jz\" (UID: \"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c\") " pod="openshift-insights/insights-operator-7c4986d449-464jz" Nov 02 22:47:54.260214 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.260186 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db94cbdb-79e6-4d1d-bf66-269fbab94abe-trusted-ca\") pod \"image-registry-86798bc874-lbrfv\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:47:54.260260 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.260242 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dccc35b-6206-47f3-b901-bafc5f17b8ad-service-ca-bundle\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:47:54.260319 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.260277 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-certificates\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:47:54.260357 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.260315 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-tmp\") pod \"insights-operator-7c4986d449-464jz\" (UID: \"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c\") " pod="openshift-insights/insights-operator-7c4986d449-464jz" Nov 02 22:47:54.260357 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.260334 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1dccc35b-6206-47f3-b901-bafc5f17b8ad-service-ca-bundle podName:1dccc35b-6206-47f3-b901-bafc5f17b8ad nodeName:}" failed. No retries permitted until 2026-11-02 22:47:54.760314937 +0000 UTC m=+33.322678275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1dccc35b-6206-47f3-b901-bafc5f17b8ad-service-ca-bundle") pod "router-default-566bb4455-kq2j4" (UID: "1dccc35b-6206-47f3-b901-bafc5f17b8ad") : configmap references non-existent config key: service-ca.crt Nov 02 22:47:54.260441 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.260366 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-464jz\" (UID: \"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c\") " pod="openshift-insights/insights-operator-7c4986d449-464jz" Nov 02 22:47:54.260441 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.260395 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-ca-trust-extracted\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:47:54.260441 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.260427 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvp6g\" (UniqueName: \"kubernetes.io/projected/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-kube-api-access-zvp6g\") pod \"insights-operator-7c4986d449-464jz\" (UID: \"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c\") " pod="openshift-insights/insights-operator-7c4986d449-464jz" Nov 02 22:47:54.260534 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.260451 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5-config\") pod \"console-operator-7cd56b8dbc-fj4n7\" (UID: \"a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" Nov 02 22:47:54.260534 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.260480 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-serving-cert\") pod \"insights-operator-7c4986d449-464jz\" (UID: \"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c\") " pod="openshift-insights/insights-operator-7c4986d449-464jz" Nov 02 22:47:54.260534 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.260507 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rf5x6\" (UID: \"2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rf5x6" Nov 02 22:47:54.260615 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.260533 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/db94cbdb-79e6-4d1d-bf66-269fbab94abe-ca-trust-extracted\") pod \"image-registry-86798bc874-lbrfv\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:47:54.260615 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.260533 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-tmp\") pod \"insights-operator-7c4986d449-464jz\" (UID: \"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c\") " pod="openshift-insights/insights-operator-7c4986d449-464jz" Nov 02 22:47:54.260615 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.260587 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-default-certificate\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:47:54.260697 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.260619 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh6h5\" (UniqueName: \"kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-kube-api-access-gh6h5\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:47:54.260697 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.260651 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-metrics-certs\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:47:54.260697 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.260676 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rn6f\" (UniqueName: \"kubernetes.io/projected/071262de-534b-4d01-bc20-784db9852f69-kube-api-access-7rn6f\") pod \"service-ca-operator-544d749c8c-xglvd\" (UID: \"071262de-534b-4d01-bc20-784db9852f69\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xglvd" Nov 02 22:47:54.260887 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.260711 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nj7zh\" (UID: \"d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nj7zh" Nov 02 22:47:54.260887 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.260740 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/071262de-534b-4d01-bc20-784db9852f69-config\") pod \"service-ca-operator-544d749c8c-xglvd\" (UID: \"071262de-534b-4d01-bc20-784db9852f69\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xglvd" Nov 02 22:47:54.260887 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.260791 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4d65\" (UniqueName: \"kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-kube-api-access-m4d65\") pod \"image-registry-86798bc874-lbrfv\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:47:54.260887 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.260821 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-image-registry-private-configuration\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:47:54.260887 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.260845 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5-trusted-ca\") pod \"console-operator-7cd56b8dbc-fj4n7\" (UID: \"a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" Nov 02 22:47:54.261560 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.261194 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 22:47:54.261560 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.261253 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-metrics-certs podName:1dccc35b-6206-47f3-b901-bafc5f17b8ad nodeName:}" failed. No retries permitted until 2026-11-02 22:47:54.761236336 +0000 UTC m=+33.323599693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-metrics-certs") pod "router-default-566bb4455-kq2j4" (UID: "1dccc35b-6206-47f3-b901-bafc5f17b8ad") : secret "router-metrics-certs-default" not found Nov 02 22:47:54.261560 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.261530 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5-config\") pod \"console-operator-7cd56b8dbc-fj4n7\" (UID: \"a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" Nov 02 22:47:54.262222 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.262191 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-464jz\" (UID: \"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c\") " pod="openshift-insights/insights-operator-7c4986d449-464jz" Nov 02 22:47:54.262632 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.262461 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5-serving-cert\") pod \"console-operator-7cd56b8dbc-fj4n7\" (UID: \"a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" Nov 02 22:47:54.264188 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.264169 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-serving-cert\") pod \"insights-operator-7c4986d449-464jz\" (UID: \"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c\") " pod="openshift-insights/insights-operator-7c4986d449-464jz" Nov 02 22:47:54.264371 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.264356 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-default-certificate\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:47:54.266802 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.266780 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-stats-auth\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:47:54.290013 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.289960 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9g8qt\"" Nov 02 22:47:54.290013 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.289956 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 22:47:54.290013 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.290013 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 22:47:54.290254 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.290012 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-cz5vj\"" Nov 02 22:47:54.290254 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.290242 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qf97t\"" Nov 02 22:47:54.311696 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.311484 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt" Nov 02 22:47:54.321727 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.321697 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfz5j\" (UniqueName: \"kubernetes.io/projected/a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5-kube-api-access-bfz5j\") pod \"console-operator-7cd56b8dbc-fj4n7\" (UID: \"a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" Nov 02 22:47:54.321946 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.321919 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjlw7\" (UniqueName: \"kubernetes.io/projected/ca5741f4-603c-448e-9088-0543eef08544-kube-api-access-bjlw7\") pod \"volume-data-source-validator-58c9f5f46d-5ljqt\" (UID: \"ca5741f4-603c-448e-9088-0543eef08544\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5ljqt" Nov 02 22:47:54.322040 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.321984 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5bg9\" (UniqueName: \"kubernetes.io/projected/1dccc35b-6206-47f3-b901-bafc5f17b8ad-kube-api-access-s5bg9\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:47:54.322331 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.322314 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvp6g\" (UniqueName: \"kubernetes.io/projected/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-kube-api-access-zvp6g\") pod \"insights-operator-7c4986d449-464jz\" (UID: \"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c\") " pod="openshift-insights/insights-operator-7c4986d449-464jz" Nov 02 22:47:54.323303 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.323287 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 22:47:54.323398 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.323364 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 22:47:54.323435 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.323410 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 22:47:54.332378 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.332357 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" Nov 02 22:47:54.356987 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.356960 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-464jz" Nov 02 22:47:54.361660 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.361625 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-tls\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:47:54.361759 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.361676 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-trusted-ca\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:47:54.361759 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.361707 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qczmw\" (UniqueName: \"kubernetes.io/projected/72ef59a1-3734-47b7-9074-f683ca94dd32-kube-api-access-qczmw\") pod \"dns-default-c8kg9\" (UID: \"72ef59a1-3734-47b7-9074-f683ca94dd32\") " pod="openshift-dns/dns-default-c8kg9" Nov 02 22:47:54.361759 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.361738 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-installation-pull-secrets\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:47:54.361916 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.361772 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/071262de-534b-4d01-bc20-784db9852f69-serving-cert\") pod \"service-ca-operator-544d749c8c-xglvd\" (UID: \"071262de-534b-4d01-bc20-784db9852f69\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xglvd" Nov 02 22:47:54.361916 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.361798 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6cmc\" (UniqueName: \"kubernetes.io/projected/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-kube-api-access-t6cmc\") pod \"cluster-samples-operator-6ddb6c8b94-rf5x6\" (UID: \"2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rf5x6" Nov 02 22:47:54.361916 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.361826 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72ef59a1-3734-47b7-9074-f683ca94dd32-metrics-tls\") pod \"dns-default-c8kg9\" (UID: \"72ef59a1-3734-47b7-9074-f683ca94dd32\") " pod="openshift-dns/dns-default-c8kg9" Nov 02 22:47:54.361916 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.361851 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwmn4\" (UniqueName: \"kubernetes.io/projected/215a42f8-881d-4cc4-b670-ea9f4aefb80b-kube-api-access-nwmn4\") pod \"ingress-canary-6jd4b\" (UID: \"215a42f8-881d-4cc4-b670-ea9f4aefb80b\") " pod="openshift-ingress-canary/ingress-canary-6jd4b" Nov 02 22:47:54.361916 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.361882 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgmgr\" (UniqueName: \"kubernetes.io/projected/ff5ad4d1-4fd9-4374-b11c-78596c9cafa8-kube-api-access-cgmgr\") pod \"network-check-source-6f78d59f99-nbswd\" (UID: \"ff5ad4d1-4fd9-4374-b11c-78596c9cafa8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nbswd" Nov 02 22:47:54.361916 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.361909 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-bound-sa-token\") pod \"image-registry-86798bc874-lbrfv\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:47:54.362247 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.361959 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-bound-sa-token\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:47:54.362247 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.361992 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/db94cbdb-79e6-4d1d-bf66-269fbab94abe-image-registry-private-configuration\") pod \"image-registry-86798bc874-lbrfv\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:47:54.362247 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.362023 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/db94cbdb-79e6-4d1d-bf66-269fbab94abe-registry-certificates\") pod \"image-registry-86798bc874-lbrfv\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:47:54.362247 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.362069 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/db94cbdb-79e6-4d1d-bf66-269fbab94abe-installation-pull-secrets\") pod \"image-registry-86798bc874-lbrfv\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:47:54.362247 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.362104 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542-nginx-conf\") pod \"networking-console-plugin-86c478bb88-nj7zh\" (UID: \"d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nj7zh" Nov 02 22:47:54.362247 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.362148 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db94cbdb-79e6-4d1d-bf66-269fbab94abe-trusted-ca\") pod \"image-registry-86798bc874-lbrfv\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:47:54.362247 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.362188 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-certificates\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:47:54.362247 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.362233 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-ca-trust-extracted\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:47:54.362608 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.362263 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/215a42f8-881d-4cc4-b670-ea9f4aefb80b-cert\") pod \"ingress-canary-6jd4b\" (UID: \"215a42f8-881d-4cc4-b670-ea9f4aefb80b\") " pod="openshift-ingress-canary/ingress-canary-6jd4b" Nov 02 22:47:54.362608 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.362303 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rf5x6\" (UID: \"2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rf5x6" Nov 02 22:47:54.362608 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.362332 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/db94cbdb-79e6-4d1d-bf66-269fbab94abe-ca-trust-extracted\") pod \"image-registry-86798bc874-lbrfv\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:47:54.362608 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.362362 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh6h5\" (UniqueName: \"kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-kube-api-access-gh6h5\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:47:54.362608 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.362401 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rn6f\" (UniqueName: \"kubernetes.io/projected/071262de-534b-4d01-bc20-784db9852f69-kube-api-access-7rn6f\") pod \"service-ca-operator-544d749c8c-xglvd\" (UID: \"071262de-534b-4d01-bc20-784db9852f69\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xglvd" Nov 02 22:47:54.362608 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.362431 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nj7zh\" (UID: \"d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nj7zh" Nov 02 22:47:54.362608 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.362460 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/071262de-534b-4d01-bc20-784db9852f69-config\") pod \"service-ca-operator-544d749c8c-xglvd\" (UID: \"071262de-534b-4d01-bc20-784db9852f69\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xglvd" Nov 02 22:47:54.362608 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.362491 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/72ef59a1-3734-47b7-9074-f683ca94dd32-config-volume\") pod \"dns-default-c8kg9\" (UID: \"72ef59a1-3734-47b7-9074-f683ca94dd32\") " pod="openshift-dns/dns-default-c8kg9" Nov 02 22:47:54.362608 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.362516 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72ef59a1-3734-47b7-9074-f683ca94dd32-tmp-dir\") pod \"dns-default-c8kg9\" (UID: \"72ef59a1-3734-47b7-9074-f683ca94dd32\") " pod="openshift-dns/dns-default-c8kg9" Nov 02 22:47:54.362608 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.362563 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4d65\" (UniqueName: \"kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-kube-api-access-m4d65\") pod \"image-registry-86798bc874-lbrfv\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:47:54.362608 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.362597 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-image-registry-private-configuration\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:47:54.362980 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.362634 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-registry-tls\") pod \"image-registry-86798bc874-lbrfv\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:47:54.362980 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.362681 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-rf5x6\" (UID: \"2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rf5x6" Nov 02 22:47:54.363482 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.363458 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-rf5x6\" (UID: \"2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rf5x6" Nov 02 22:47:54.363607 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.363590 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 22:47:54.363643 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.363610 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bc8cbc8c-hghqr: secret "image-registry-tls" not found Nov 02 22:47:54.363675 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.363664 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-tls podName:2d578e40-60d8-4ebd-b0b5-2f5c274c7903 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:54.863644516 +0000 UTC m=+33.426007850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-tls") pod "image-registry-bc8cbc8c-hghqr" (UID: "2d578e40-60d8-4ebd-b0b5-2f5c274c7903") : secret "image-registry-tls" not found Nov 02 22:47:54.367988 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.364805 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-trusted-ca\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:47:54.367988 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.365154 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-ca-trust-extracted\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:47:54.367988 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.366854 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542-nginx-conf\") pod \"networking-console-plugin-86c478bb88-nj7zh\" (UID: \"d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nj7zh" Nov 02 22:47:54.367988 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.367322 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/db94cbdb-79e6-4d1d-bf66-269fbab94abe-registry-certificates\") pod \"image-registry-86798bc874-lbrfv\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:47:54.367988 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.367354 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db94cbdb-79e6-4d1d-bf66-269fbab94abe-trusted-ca\") pod \"image-registry-86798bc874-lbrfv\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:47:54.367988 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.367464 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 22:47:54.367988 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.367522 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542-networking-console-plugin-cert podName:d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:54.867502915 +0000 UTC m=+33.429866244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-nj7zh" (UID: "d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542") : secret "networking-console-plugin-cert" not found Nov 02 22:47:54.367988 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.367612 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 22:47:54.367988 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.367646 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-samples-operator-tls podName:2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc nodeName:}" failed. No retries permitted until 2026-11-02 22:47:54.867634773 +0000 UTC m=+33.429998114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-rf5x6" (UID: "2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc") : secret "samples-operator-tls" not found Nov 02 22:47:54.367988 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.367666 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-installation-pull-secrets\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:47:54.368515 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.368227 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/db94cbdb-79e6-4d1d-bf66-269fbab94abe-ca-trust-extracted\") pod \"image-registry-86798bc874-lbrfv\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:47:54.368515 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.368232 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-certificates\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:47:54.368515 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.368340 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 22:47:54.368515 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.368354 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86798bc874-lbrfv: secret "image-registry-tls" not found Nov 02 22:47:54.368515 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.368395 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-registry-tls podName:db94cbdb-79e6-4d1d-bf66-269fbab94abe nodeName:}" failed. No retries permitted until 2026-11-02 22:47:54.868380947 +0000 UTC m=+33.430744281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-registry-tls") pod "image-registry-86798bc874-lbrfv" (UID: "db94cbdb-79e6-4d1d-bf66-269fbab94abe") : secret "image-registry-tls" not found Nov 02 22:47:54.368742 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.368720 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/071262de-534b-4d01-bc20-784db9852f69-serving-cert\") pod \"service-ca-operator-544d749c8c-xglvd\" (UID: \"071262de-534b-4d01-bc20-784db9852f69\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xglvd" Nov 02 22:47:54.368858 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.368839 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/071262de-534b-4d01-bc20-784db9852f69-config\") pod \"service-ca-operator-544d749c8c-xglvd\" (UID: \"071262de-534b-4d01-bc20-784db9852f69\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xglvd" Nov 02 22:47:54.369476 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.369453 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/db94cbdb-79e6-4d1d-bf66-269fbab94abe-image-registry-private-configuration\") pod \"image-registry-86798bc874-lbrfv\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:47:54.374521 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.374225 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-image-registry-private-configuration\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:47:54.375191 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.375167 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/db94cbdb-79e6-4d1d-bf66-269fbab94abe-installation-pull-secrets\") pod \"image-registry-86798bc874-lbrfv\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:47:54.391452 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.391307 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5ljqt" Nov 02 22:47:54.427043 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.426665 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-bound-sa-token\") pod \"image-registry-86798bc874-lbrfv\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:47:54.427221 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.427193 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-bound-sa-token\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:47:54.430692 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.427403 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6cmc\" (UniqueName: \"kubernetes.io/projected/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-kube-api-access-t6cmc\") pod \"cluster-samples-operator-6ddb6c8b94-rf5x6\" (UID: \"2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rf5x6" Nov 02 22:47:54.430692 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.427811 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgmgr\" (UniqueName: \"kubernetes.io/projected/ff5ad4d1-4fd9-4374-b11c-78596c9cafa8-kube-api-access-cgmgr\") pod \"network-check-source-6f78d59f99-nbswd\" (UID: \"ff5ad4d1-4fd9-4374-b11c-78596c9cafa8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nbswd" Nov 02 22:47:54.430692 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.429517 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rn6f\" (UniqueName: \"kubernetes.io/projected/071262de-534b-4d01-bc20-784db9852f69-kube-api-access-7rn6f\") pod \"service-ca-operator-544d749c8c-xglvd\" (UID: \"071262de-534b-4d01-bc20-784db9852f69\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xglvd" Nov 02 22:47:54.431090 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.430917 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4d65\" (UniqueName: \"kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-kube-api-access-m4d65\") pod \"image-registry-86798bc874-lbrfv\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:47:54.432917 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.432890 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh6h5\" (UniqueName: \"kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-kube-api-access-gh6h5\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:47:54.464251 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.463443 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/215a42f8-881d-4cc4-b670-ea9f4aefb80b-cert\") pod \"ingress-canary-6jd4b\" (UID: \"215a42f8-881d-4cc4-b670-ea9f4aefb80b\") " pod="openshift-ingress-canary/ingress-canary-6jd4b" Nov 02 22:47:54.464251 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.463543 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/72ef59a1-3734-47b7-9074-f683ca94dd32-config-volume\") pod \"dns-default-c8kg9\" (UID: \"72ef59a1-3734-47b7-9074-f683ca94dd32\") " pod="openshift-dns/dns-default-c8kg9" Nov 02 22:47:54.464251 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.463569 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72ef59a1-3734-47b7-9074-f683ca94dd32-tmp-dir\") pod \"dns-default-c8kg9\" (UID: \"72ef59a1-3734-47b7-9074-f683ca94dd32\") " pod="openshift-dns/dns-default-c8kg9" Nov 02 22:47:54.464251 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.463659 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qczmw\" (UniqueName: \"kubernetes.io/projected/72ef59a1-3734-47b7-9074-f683ca94dd32-kube-api-access-qczmw\") pod \"dns-default-c8kg9\" (UID: \"72ef59a1-3734-47b7-9074-f683ca94dd32\") " pod="openshift-dns/dns-default-c8kg9" Nov 02 22:47:54.464251 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.463697 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72ef59a1-3734-47b7-9074-f683ca94dd32-metrics-tls\") pod \"dns-default-c8kg9\" (UID: \"72ef59a1-3734-47b7-9074-f683ca94dd32\") " pod="openshift-dns/dns-default-c8kg9" Nov 02 22:47:54.464251 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.463722 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwmn4\" (UniqueName: \"kubernetes.io/projected/215a42f8-881d-4cc4-b670-ea9f4aefb80b-kube-api-access-nwmn4\") pod \"ingress-canary-6jd4b\" (UID: \"215a42f8-881d-4cc4-b670-ea9f4aefb80b\") " pod="openshift-ingress-canary/ingress-canary-6jd4b" Nov 02 22:47:54.464251 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.464159 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 22:47:54.464251 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.464225 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72ef59a1-3734-47b7-9074-f683ca94dd32-metrics-tls podName:72ef59a1-3734-47b7-9074-f683ca94dd32 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:54.964205397 +0000 UTC m=+33.526568715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72ef59a1-3734-47b7-9074-f683ca94dd32-metrics-tls") pod "dns-default-c8kg9" (UID: "72ef59a1-3734-47b7-9074-f683ca94dd32") : secret "dns-default-metrics-tls" not found Nov 02 22:47:54.464601 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.464509 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/72ef59a1-3734-47b7-9074-f683ca94dd32-config-volume\") pod \"dns-default-c8kg9\" (UID: \"72ef59a1-3734-47b7-9074-f683ca94dd32\") " pod="openshift-dns/dns-default-c8kg9" Nov 02 22:47:54.464749 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.464728 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72ef59a1-3734-47b7-9074-f683ca94dd32-tmp-dir\") pod \"dns-default-c8kg9\" (UID: \"72ef59a1-3734-47b7-9074-f683ca94dd32\") " pod="openshift-dns/dns-default-c8kg9" Nov 02 22:47:54.466295 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.466275 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/215a42f8-881d-4cc4-b670-ea9f4aefb80b-cert\") pod \"ingress-canary-6jd4b\" (UID: \"215a42f8-881d-4cc4-b670-ea9f4aefb80b\") " pod="openshift-ingress-canary/ingress-canary-6jd4b" Nov 02 22:47:54.517697 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.517672 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nbswd" Nov 02 22:47:54.523707 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.523681 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qczmw\" (UniqueName: \"kubernetes.io/projected/72ef59a1-3734-47b7-9074-f683ca94dd32-kube-api-access-qczmw\") pod \"dns-default-c8kg9\" (UID: \"72ef59a1-3734-47b7-9074-f683ca94dd32\") " pod="openshift-dns/dns-default-c8kg9" Nov 02 22:47:54.523934 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.523914 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwmn4\" (UniqueName: \"kubernetes.io/projected/215a42f8-881d-4cc4-b670-ea9f4aefb80b-kube-api-access-nwmn4\") pod \"ingress-canary-6jd4b\" (UID: \"215a42f8-881d-4cc4-b670-ea9f4aefb80b\") " pod="openshift-ingress-canary/ingress-canary-6jd4b" Nov 02 22:47:54.575996 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.575962 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6jd4b" Nov 02 22:47:54.665695 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.665669 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ba063f-0db3-4f82-b280-40718c28b479-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qcz4m\" (UID: \"53ba063f-0db3-4f82-b280-40718c28b479\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qcz4m" Nov 02 22:47:54.665889 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.665837 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 22:47:54.665960 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.665920 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ba063f-0db3-4f82-b280-40718c28b479-cluster-monitoring-operator-tls podName:53ba063f-0db3-4f82-b280-40718c28b479 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:55.665898556 +0000 UTC m=+34.228261876 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/53ba063f-0db3-4f82-b280-40718c28b479-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qcz4m" (UID: "53ba063f-0db3-4f82-b280-40718c28b479") : secret "cluster-monitoring-operator-tls" not found Nov 02 22:47:54.673748 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.673670 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5wnxz"] Nov 02 22:47:54.676898 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.676870 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-25msk"] Nov 02 22:47:54.676993 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:54.676907 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d56c63_207c_42b3_8946_c993d0fde66b.slice/crio-1dbff91d6a416a7a712a6415f02a9e2f4fb91cbbeeacad097a2516b6781d2736 WatchSource:0}: Error finding container 1dbff91d6a416a7a712a6415f02a9e2f4fb91cbbeeacad097a2516b6781d2736: Status 404 returned error can't find the container with id 1dbff91d6a416a7a712a6415f02a9e2f4fb91cbbeeacad097a2516b6781d2736 Nov 02 22:47:54.677989 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.677970 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-276wh"] Nov 02 22:47:54.681895 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:54.680237 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecdc26b5_be99_4928_8fff_203cc2c19ad6.slice/crio-a1182eba2e295734048a99282d575c2778a1cf825dcce39ca7ad71baf84623de WatchSource:0}: Error finding container a1182eba2e295734048a99282d575c2778a1cf825dcce39ca7ad71baf84623de: Status 404 returned error can't find the container with id a1182eba2e295734048a99282d575c2778a1cf825dcce39ca7ad71baf84623de Nov 02 22:47:54.714009 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.713779 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xglvd" Nov 02 22:47:54.727796 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.727763 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt"] Nov 02 22:47:54.736062 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:54.736020 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae96a9c1_10a6_4050_bca0_6a2e7859d3c3.slice/crio-e5bd2e832314848947277f81dc2403220f6b50e84660c42dd50d4371840a7200 WatchSource:0}: Error finding container e5bd2e832314848947277f81dc2403220f6b50e84660c42dd50d4371840a7200: Status 404 returned error can't find the container with id e5bd2e832314848947277f81dc2403220f6b50e84660c42dd50d4371840a7200 Nov 02 22:47:54.754875 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.754702 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-fj4n7"] Nov 02 22:47:54.761367 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:54.761328 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5cdd7b8_bc10_4788_a37a_2f93c2fbe0a5.slice/crio-6002186f05dc05c6ae18544edac28231f79c6e9d14c1d0c302170d8d3844800f WatchSource:0}: Error finding container 6002186f05dc05c6ae18544edac28231f79c6e9d14c1d0c302170d8d3844800f: Status 404 returned error can't find the container with id 6002186f05dc05c6ae18544edac28231f79c6e9d14c1d0c302170d8d3844800f Nov 02 22:47:54.766802 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.766469 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-metrics-certs\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:47:54.766802 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.766636 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 22:47:54.766802 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.766658 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dccc35b-6206-47f3-b901-bafc5f17b8ad-service-ca-bundle\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:47:54.766802 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.766699 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-metrics-certs podName:1dccc35b-6206-47f3-b901-bafc5f17b8ad nodeName:}" failed. No retries permitted until 2026-11-02 22:47:55.766677885 +0000 UTC m=+34.329041221 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-metrics-certs") pod "router-default-566bb4455-kq2j4" (UID: "1dccc35b-6206-47f3-b901-bafc5f17b8ad") : secret "router-metrics-certs-default" not found Nov 02 22:47:54.766802 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.766776 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1dccc35b-6206-47f3-b901-bafc5f17b8ad-service-ca-bundle podName:1dccc35b-6206-47f3-b901-bafc5f17b8ad nodeName:}" failed. No retries permitted until 2026-11-02 22:47:55.766757849 +0000 UTC m=+34.329121177 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1dccc35b-6206-47f3-b901-bafc5f17b8ad-service-ca-bundle") pod "router-default-566bb4455-kq2j4" (UID: "1dccc35b-6206-47f3-b901-bafc5f17b8ad") : configmap references non-existent config key: service-ca.crt Nov 02 22:47:54.774480 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.773723 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-464jz"] Nov 02 22:47:54.780017 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:54.779886 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb8c24bf_0fc9_4d8b_84a6_2fc5d652950c.slice/crio-4e20810858ab32fdce1dd75df90c3dbfc2084c6aca8fa0711e884db900522380 WatchSource:0}: Error finding container 4e20810858ab32fdce1dd75df90c3dbfc2084c6aca8fa0711e884db900522380: Status 404 returned error can't find the container with id 4e20810858ab32fdce1dd75df90c3dbfc2084c6aca8fa0711e884db900522380 Nov 02 22:47:54.805115 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.804294 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5ljqt"] Nov 02 22:47:54.817910 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:54.817077 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca5741f4_603c_448e_9088_0543eef08544.slice/crio-b8fc48b29d7b468edaf2b1f8982de2890976ae02d3e5e3d7a4613957c394f5a0 WatchSource:0}: Error finding container b8fc48b29d7b468edaf2b1f8982de2890976ae02d3e5e3d7a4613957c394f5a0: Status 404 returned error can't find the container with id b8fc48b29d7b468edaf2b1f8982de2890976ae02d3e5e3d7a4613957c394f5a0 Nov 02 22:47:54.868010 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.867979 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nj7zh\" (UID: \"d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nj7zh" Nov 02 22:47:54.868164 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.868024 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/417d26e1-2a9d-41e3-8940-69626662a2f3-original-pull-secret\") pod \"global-pull-secret-syncer-wbxxp\" (UID: \"417d26e1-2a9d-41e3-8940-69626662a2f3\") " pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:47:54.868164 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.868119 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:54.868164 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.868153 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 22:47:54.868307 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.868183 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/417d26e1-2a9d-41e3-8940-69626662a2f3-original-pull-secret podName:417d26e1-2a9d-41e3-8940-69626662a2f3 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:26.868171343 +0000 UTC m=+65.430534657 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/417d26e1-2a9d-41e3-8940-69626662a2f3-original-pull-secret") pod "global-pull-secret-syncer-wbxxp" (UID: "417d26e1-2a9d-41e3-8940-69626662a2f3") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:54.868307 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.868218 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542-networking-console-plugin-cert podName:d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:55.868199331 +0000 UTC m=+34.430562660 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-nj7zh" (UID: "d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542") : secret "networking-console-plugin-cert" not found Nov 02 22:47:54.868307 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.868282 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-tls\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:47:54.868461 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.868386 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82b3b5e2-09f1-49c0-8d72-c85187918223-metrics-certs\") pod \"network-metrics-daemon-465bf\" (UID: \"82b3b5e2-09f1-49c0-8d72-c85187918223\") " pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:47:54.868461 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.868393 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 22:47:54.868461 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.868408 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bc8cbc8c-hghqr: secret "image-registry-tls" not found Nov 02 22:47:54.868461 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.868443 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-tls podName:2d578e40-60d8-4ebd-b0b5-2f5c274c7903 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:55.868431933 +0000 UTC m=+34.430795252 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-tls") pod "image-registry-bc8cbc8c-hghqr" (UID: "2d578e40-60d8-4ebd-b0b5-2f5c274c7903") : secret "image-registry-tls" not found Nov 02 22:47:54.868615 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.868481 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rf5x6\" (UID: \"2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rf5x6" Nov 02 22:47:54.868615 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.868499 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:54.868615 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.868541 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82b3b5e2-09f1-49c0-8d72-c85187918223-metrics-certs podName:82b3b5e2-09f1-49c0-8d72-c85187918223 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:26.868526482 +0000 UTC m=+65.430889812 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82b3b5e2-09f1-49c0-8d72-c85187918223-metrics-certs") pod "network-metrics-daemon-465bf" (UID: "82b3b5e2-09f1-49c0-8d72-c85187918223") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:54.868615 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.868555 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 22:47:54.868615 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.868585 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-samples-operator-tls podName:2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc nodeName:}" failed. No retries permitted until 2026-11-02 22:47:55.868577452 +0000 UTC m=+34.430940766 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-rf5x6" (UID: "2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc") : secret "samples-operator-tls" not found Nov 02 22:47:54.898611 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.898551 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-nbswd"] Nov 02 22:47:54.901148 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:54.901112 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff5ad4d1_4fd9_4374_b11c_78596c9cafa8.slice/crio-2f5f80417d312f9989700350646e7b60429b5eb784cc4e0e275da19371dbc8d3 WatchSource:0}: Error finding container 2f5f80417d312f9989700350646e7b60429b5eb784cc4e0e275da19371dbc8d3: Status 404 returned error can't find the container with id 2f5f80417d312f9989700350646e7b60429b5eb784cc4e0e275da19371dbc8d3 Nov 02 22:47:54.917985 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.917962 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:47:54.918077 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.918002 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:47:54.969171 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.969114 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72ef59a1-3734-47b7-9074-f683ca94dd32-metrics-tls\") pod \"dns-default-c8kg9\" (UID: \"72ef59a1-3734-47b7-9074-f683ca94dd32\") " pod="openshift-dns/dns-default-c8kg9" Nov 02 22:47:54.969312 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.969285 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-registry-tls\") pod \"image-registry-86798bc874-lbrfv\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:47:54.969312 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.969305 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqn4t\" (UniqueName: \"kubernetes.io/projected/566b2dae-dd3e-4c63-b4d2-e30667a5fd07-kube-api-access-cqn4t\") pod \"network-check-target-2qpkq\" (UID: \"566b2dae-dd3e-4c63-b4d2-e30667a5fd07\") " pod="openshift-network-diagnostics/network-check-target-2qpkq" Nov 02 22:47:54.969382 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.969305 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 22:47:54.969417 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.969385 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 22:47:54.969417 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.969402 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86798bc874-lbrfv: secret "image-registry-tls" not found Nov 02 22:47:54.969486 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.969392 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72ef59a1-3734-47b7-9074-f683ca94dd32-metrics-tls podName:72ef59a1-3734-47b7-9074-f683ca94dd32 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:55.969379975 +0000 UTC m=+34.531743289 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72ef59a1-3734-47b7-9074-f683ca94dd32-metrics-tls") pod "dns-default-c8kg9" (UID: "72ef59a1-3734-47b7-9074-f683ca94dd32") : secret "dns-default-metrics-tls" not found Nov 02 22:47:54.969486 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:54.969460 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-registry-tls podName:db94cbdb-79e6-4d1d-bf66-269fbab94abe nodeName:}" failed. No retries permitted until 2026-11-02 22:47:55.969444068 +0000 UTC m=+34.531807389 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-registry-tls") pod "image-registry-86798bc874-lbrfv" (UID: "db94cbdb-79e6-4d1d-bf66-269fbab94abe") : secret "image-registry-tls" not found Nov 02 22:47:54.970229 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.970210 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 22:47:54.970297 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.970271 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 22:47:54.970339 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.970317 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zfnv8\"" Nov 02 22:47:54.970370 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.970351 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r8wwr\"" Nov 02 22:47:54.971772 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:54.971754 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqn4t\" (UniqueName: \"kubernetes.io/projected/566b2dae-dd3e-4c63-b4d2-e30667a5fd07-kube-api-access-cqn4t\") pod \"network-check-target-2qpkq\" (UID: \"566b2dae-dd3e-4c63-b4d2-e30667a5fd07\") " pod="openshift-network-diagnostics/network-check-target-2qpkq" Nov 02 22:47:55.073225 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:55.073197 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6jd4b"] Nov 02 22:47:55.076052 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:55.076020 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod215a42f8_881d_4cc4_b670_ea9f4aefb80b.slice/crio-4a29976e47440460e64c2e170cbc823f58c5efb479a898fb03278815767a5c2b WatchSource:0}: Error finding container 4a29976e47440460e64c2e170cbc823f58c5efb479a898fb03278815767a5c2b: Status 404 returned error can't find the container with id 4a29976e47440460e64c2e170cbc823f58c5efb479a898fb03278815767a5c2b Nov 02 22:47:55.111440 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:55.111408 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nbswd" event={"ID":"ff5ad4d1-4fd9-4374-b11c-78596c9cafa8","Type":"ContainerStarted","Data":"2f5f80417d312f9989700350646e7b60429b5eb784cc4e0e275da19371dbc8d3"} Nov 02 22:47:55.112630 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:55.112603 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-464jz" event={"ID":"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c","Type":"ContainerStarted","Data":"4e20810858ab32fdce1dd75df90c3dbfc2084c6aca8fa0711e884db900522380"} Nov 02 22:47:55.113773 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:55.113749 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" event={"ID":"a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5","Type":"ContainerStarted","Data":"6002186f05dc05c6ae18544edac28231f79c6e9d14c1d0c302170d8d3844800f"} Nov 02 22:47:55.114942 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:55.114921 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5ljqt" event={"ID":"ca5741f4-603c-448e-9088-0543eef08544","Type":"ContainerStarted","Data":"b8fc48b29d7b468edaf2b1f8982de2890976ae02d3e5e3d7a4613957c394f5a0"} Nov 02 22:47:55.116266 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:55.116241 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6jd4b" event={"ID":"215a42f8-881d-4cc4-b670-ea9f4aefb80b","Type":"ContainerStarted","Data":"4a29976e47440460e64c2e170cbc823f58c5efb479a898fb03278815767a5c2b"} Nov 02 22:47:55.117256 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:55.117235 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt" event={"ID":"ae96a9c1-10a6-4050-bca0-6a2e7859d3c3","Type":"ContainerStarted","Data":"e5bd2e832314848947277f81dc2403220f6b50e84660c42dd50d4371840a7200"} Nov 02 22:47:55.118613 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:55.118593 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-276wh" event={"ID":"12ce41df-43ef-4b00-be8e-5d3fad591eea","Type":"ContainerStarted","Data":"f76bfbfb1510d67d54a1aff069b465423e16dffe1066d9f78b74e13e24d510ff"} Nov 02 22:47:55.118690 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:55.118626 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-276wh" event={"ID":"12ce41df-43ef-4b00-be8e-5d3fad591eea","Type":"ContainerStarted","Data":"320f557ca944b584051436e7f2df20c9bdd300bcbb801c8a193fc912ffec0505"} Nov 02 22:47:55.119988 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:55.119969 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5wnxz" event={"ID":"a8d56c63-207c-42b3-8946-c993d0fde66b","Type":"ContainerStarted","Data":"d92816a5a5ee5896204d93fa4331383b218bb827d07901a634bc74f65f4d4266"} Nov 02 22:47:55.120078 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:55.120003 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5wnxz" event={"ID":"a8d56c63-207c-42b3-8946-c993d0fde66b","Type":"ContainerStarted","Data":"1dbff91d6a416a7a712a6415f02a9e2f4fb91cbbeeacad097a2516b6781d2736"} Nov 02 22:47:55.121447 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:55.121428 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-25msk" event={"ID":"ecdc26b5-be99-4928-8fff-203cc2c19ad6","Type":"ContainerStarted","Data":"ed7e31445aa4d40f2283c04d7d48c90ba2bcd3c4ed0e2521aa9696fda5b4e0a6"} Nov 02 22:47:55.121524 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:55.121456 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-25msk" event={"ID":"ecdc26b5-be99-4928-8fff-203cc2c19ad6","Type":"ContainerStarted","Data":"a1182eba2e295734048a99282d575c2778a1cf825dcce39ca7ad71baf84623de"} Nov 02 22:47:55.136508 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:55.136483 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-xglvd"] Nov 02 22:47:55.140388 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:55.140368 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod071262de_534b_4d01_bc20_784db9852f69.slice/crio-b4338a767aa48da2c47aa03999ce26022ebcff53ee47f2cc3b29218644d0b581 WatchSource:0}: Error finding container b4338a767aa48da2c47aa03999ce26022ebcff53ee47f2cc3b29218644d0b581: Status 404 returned error can't find the container with id b4338a767aa48da2c47aa03999ce26022ebcff53ee47f2cc3b29218644d0b581 Nov 02 22:47:55.189683 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:55.189599 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qpkq" Nov 02 22:47:55.583091 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:55.583056 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2qpkq"] Nov 02 22:47:55.585840 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:47:55.585802 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod566b2dae_dd3e_4c63_b4d2_e30667a5fd07.slice/crio-6ea9e8c1f04b4aab4006eeac81a30099666f780462a003dcf5da20294f0e0c9a WatchSource:0}: Error finding container 6ea9e8c1f04b4aab4006eeac81a30099666f780462a003dcf5da20294f0e0c9a: Status 404 returned error can't find the container with id 6ea9e8c1f04b4aab4006eeac81a30099666f780462a003dcf5da20294f0e0c9a Nov 02 22:47:55.689237 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:55.689202 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ba063f-0db3-4f82-b280-40718c28b479-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qcz4m\" (UID: \"53ba063f-0db3-4f82-b280-40718c28b479\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qcz4m" Nov 02 22:47:55.689400 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:55.689366 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 22:47:55.689459 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:55.689431 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ba063f-0db3-4f82-b280-40718c28b479-cluster-monitoring-operator-tls podName:53ba063f-0db3-4f82-b280-40718c28b479 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:57.689413723 +0000 UTC m=+36.251777044 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/53ba063f-0db3-4f82-b280-40718c28b479-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qcz4m" (UID: "53ba063f-0db3-4f82-b280-40718c28b479") : secret "cluster-monitoring-operator-tls" not found Nov 02 22:47:55.790030 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:55.789994 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-metrics-certs\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:47:55.790483 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:55.790273 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dccc35b-6206-47f3-b901-bafc5f17b8ad-service-ca-bundle\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:47:55.790483 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:55.790434 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1dccc35b-6206-47f3-b901-bafc5f17b8ad-service-ca-bundle podName:1dccc35b-6206-47f3-b901-bafc5f17b8ad nodeName:}" failed. No retries permitted until 2026-11-02 22:47:57.790418907 +0000 UTC m=+36.352782236 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1dccc35b-6206-47f3-b901-bafc5f17b8ad-service-ca-bundle") pod "router-default-566bb4455-kq2j4" (UID: "1dccc35b-6206-47f3-b901-bafc5f17b8ad") : configmap references non-existent config key: service-ca.crt Nov 02 22:47:55.790562 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:55.790540 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 22:47:55.790595 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:55.790582 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-metrics-certs podName:1dccc35b-6206-47f3-b901-bafc5f17b8ad nodeName:}" failed. No retries permitted until 2026-11-02 22:47:57.790570165 +0000 UTC m=+36.352933497 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-metrics-certs") pod "router-default-566bb4455-kq2j4" (UID: "1dccc35b-6206-47f3-b901-bafc5f17b8ad") : secret "router-metrics-certs-default" not found Nov 02 22:47:55.891012 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:55.890925 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rf5x6\" (UID: \"2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rf5x6" Nov 02 22:47:55.891012 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:55.890996 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nj7zh\" (UID: \"d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nj7zh" Nov 02 22:47:55.891334 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:55.891068 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-tls\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:47:55.891334 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:55.891088 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 22:47:55.891334 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:55.891112 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 22:47:55.891334 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:55.891174 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-samples-operator-tls podName:2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc nodeName:}" failed. No retries permitted until 2026-11-02 22:47:57.891155422 +0000 UTC m=+36.453518754 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-rf5x6" (UID: "2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc") : secret "samples-operator-tls" not found Nov 02 22:47:55.891334 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:55.891188 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542-networking-console-plugin-cert podName:d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:57.891181723 +0000 UTC m=+36.453545037 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-nj7zh" (UID: "d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542") : secret "networking-console-plugin-cert" not found Nov 02 22:47:55.891334 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:55.891206 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 22:47:55.891334 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:55.891220 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bc8cbc8c-hghqr: secret "image-registry-tls" not found Nov 02 22:47:55.891334 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:55.891266 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-tls podName:2d578e40-60d8-4ebd-b0b5-2f5c274c7903 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:57.891251331 +0000 UTC m=+36.453614665 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-tls") pod "image-registry-bc8cbc8c-hghqr" (UID: "2d578e40-60d8-4ebd-b0b5-2f5c274c7903") : secret "image-registry-tls" not found Nov 02 22:47:55.950179 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:55.950103 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-25msk" podStartSLOduration=408.950074465 podStartE2EDuration="6m48.950074465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:55.94848726 +0000 UTC m=+34.510850620" watchObservedRunningTime="2026-11-02 22:47:55.950074465 +0000 UTC m=+34.512437803" Nov 02 22:47:55.992092 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:55.991518 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72ef59a1-3734-47b7-9074-f683ca94dd32-metrics-tls\") pod \"dns-default-c8kg9\" (UID: \"72ef59a1-3734-47b7-9074-f683ca94dd32\") " pod="openshift-dns/dns-default-c8kg9" Nov 02 22:47:55.992092 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:55.991731 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-registry-tls\") pod \"image-registry-86798bc874-lbrfv\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:47:55.992092 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:55.991882 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 22:47:55.992092 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:55.991896 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86798bc874-lbrfv: secret "image-registry-tls" not found Nov 02 22:47:55.992092 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:55.991950 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-registry-tls podName:db94cbdb-79e6-4d1d-bf66-269fbab94abe nodeName:}" failed. No retries permitted until 2026-11-02 22:47:57.991931888 +0000 UTC m=+36.554295216 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-registry-tls") pod "image-registry-86798bc874-lbrfv" (UID: "db94cbdb-79e6-4d1d-bf66-269fbab94abe") : secret "image-registry-tls" not found Nov 02 22:47:55.992092 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:55.992024 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 22:47:55.992092 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:55.992060 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72ef59a1-3734-47b7-9074-f683ca94dd32-metrics-tls podName:72ef59a1-3734-47b7-9074-f683ca94dd32 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:57.992048219 +0000 UTC m=+36.554411551 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72ef59a1-3734-47b7-9074-f683ca94dd32-metrics-tls") pod "dns-default-c8kg9" (UID: "72ef59a1-3734-47b7-9074-f683ca94dd32") : secret "dns-default-metrics-tls" not found Nov 02 22:47:56.125522 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:56.125456 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xglvd" event={"ID":"071262de-534b-4d01-bc20-784db9852f69","Type":"ContainerStarted","Data":"b4338a767aa48da2c47aa03999ce26022ebcff53ee47f2cc3b29218644d0b581"} Nov 02 22:47:56.131043 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:56.130586 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2qpkq" event={"ID":"566b2dae-dd3e-4c63-b4d2-e30667a5fd07","Type":"ContainerStarted","Data":"6ea9e8c1f04b4aab4006eeac81a30099666f780462a003dcf5da20294f0e0c9a"} Nov 02 22:47:56.180198 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:56.179311 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-276wh" podStartSLOduration=409.179283296 podStartE2EDuration="6m49.179283296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:56.178771661 +0000 UTC m=+34.741135018" watchObservedRunningTime="2026-11-02 22:47:56.179283296 +0000 UTC m=+34.741646633" Nov 02 22:47:56.180198 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:56.179670 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-5wnxz" podStartSLOduration=409.179657713 podStartE2EDuration="6m49.179657713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:56.064789714 +0000 UTC m=+34.627153055" watchObservedRunningTime="2026-11-02 22:47:56.179657713 +0000 UTC m=+34.742021051" Nov 02 22:47:57.712724 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:57.711992 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ba063f-0db3-4f82-b280-40718c28b479-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qcz4m\" (UID: \"53ba063f-0db3-4f82-b280-40718c28b479\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qcz4m" Nov 02 22:47:57.712724 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:57.712226 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 22:47:57.712724 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:57.712293 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ba063f-0db3-4f82-b280-40718c28b479-cluster-monitoring-operator-tls podName:53ba063f-0db3-4f82-b280-40718c28b479 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:01.712275145 +0000 UTC m=+40.274638474 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/53ba063f-0db3-4f82-b280-40718c28b479-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qcz4m" (UID: "53ba063f-0db3-4f82-b280-40718c28b479") : secret "cluster-monitoring-operator-tls" not found Nov 02 22:47:57.813399 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:57.813357 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dccc35b-6206-47f3-b901-bafc5f17b8ad-service-ca-bundle\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:47:57.813558 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:57.813443 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-metrics-certs\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:47:57.814238 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:57.813662 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 22:47:57.814238 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:57.813729 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-metrics-certs podName:1dccc35b-6206-47f3-b901-bafc5f17b8ad nodeName:}" failed. No retries permitted until 2026-11-02 22:48:01.813711802 +0000 UTC m=+40.376075123 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-metrics-certs") pod "router-default-566bb4455-kq2j4" (UID: "1dccc35b-6206-47f3-b901-bafc5f17b8ad") : secret "router-metrics-certs-default" not found Nov 02 22:47:57.814238 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:57.814155 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1dccc35b-6206-47f3-b901-bafc5f17b8ad-service-ca-bundle podName:1dccc35b-6206-47f3-b901-bafc5f17b8ad nodeName:}" failed. No retries permitted until 2026-11-02 22:48:01.814117045 +0000 UTC m=+40.376480375 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1dccc35b-6206-47f3-b901-bafc5f17b8ad-service-ca-bundle") pod "router-default-566bb4455-kq2j4" (UID: "1dccc35b-6206-47f3-b901-bafc5f17b8ad") : configmap references non-existent config key: service-ca.crt Nov 02 22:47:57.914649 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:57.914612 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rf5x6\" (UID: \"2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rf5x6" Nov 02 22:47:57.914820 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:57.914682 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nj7zh\" (UID: \"d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nj7zh" Nov 02 22:47:57.914820 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:57.914758 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-tls\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:47:57.915175 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:57.914946 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 22:47:57.915175 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:57.914960 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bc8cbc8c-hghqr: secret "image-registry-tls" not found Nov 02 22:47:57.915175 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:57.915005 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-tls podName:2d578e40-60d8-4ebd-b0b5-2f5c274c7903 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:01.914991646 +0000 UTC m=+40.477354960 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-tls") pod "image-registry-bc8cbc8c-hghqr" (UID: "2d578e40-60d8-4ebd-b0b5-2f5c274c7903") : secret "image-registry-tls" not found Nov 02 22:47:57.915175 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:57.915041 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 22:47:57.915175 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:57.915053 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 22:47:57.915175 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:57.915079 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542-networking-console-plugin-cert podName:d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:01.91507138 +0000 UTC m=+40.477434694 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-nj7zh" (UID: "d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542") : secret "networking-console-plugin-cert" not found Nov 02 22:47:57.915175 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:57.915121 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-samples-operator-tls podName:2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc nodeName:}" failed. No retries permitted until 2026-11-02 22:48:01.915102343 +0000 UTC m=+40.477465681 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-rf5x6" (UID: "2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc") : secret "samples-operator-tls" not found Nov 02 22:47:58.016096 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:58.016015 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72ef59a1-3734-47b7-9074-f683ca94dd32-metrics-tls\") pod \"dns-default-c8kg9\" (UID: \"72ef59a1-3734-47b7-9074-f683ca94dd32\") " pod="openshift-dns/dns-default-c8kg9" Nov 02 22:47:58.016262 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:58.016229 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-registry-tls\") pod \"image-registry-86798bc874-lbrfv\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:47:58.016426 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:58.016406 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 22:47:58.016426 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:58.016428 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86798bc874-lbrfv: secret "image-registry-tls" not found Nov 02 22:47:58.016550 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:58.016486 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-registry-tls podName:db94cbdb-79e6-4d1d-bf66-269fbab94abe nodeName:}" failed. No retries permitted until 2026-11-02 22:48:02.016468772 +0000 UTC m=+40.578832100 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-registry-tls") pod "image-registry-86798bc874-lbrfv" (UID: "db94cbdb-79e6-4d1d-bf66-269fbab94abe") : secret "image-registry-tls" not found Nov 02 22:47:58.016613 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:58.016555 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 22:47:58.016613 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:47:58.016588 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72ef59a1-3734-47b7-9074-f683ca94dd32-metrics-tls podName:72ef59a1-3734-47b7-9074-f683ca94dd32 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:02.016575276 +0000 UTC m=+40.578938619 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72ef59a1-3734-47b7-9074-f683ca94dd32-metrics-tls") pod "dns-default-c8kg9" (UID: "72ef59a1-3734-47b7-9074-f683ca94dd32") : secret "dns-default-metrics-tls" not found Nov 02 22:47:58.143090 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:58.143007 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5ljqt" event={"ID":"ca5741f4-603c-448e-9088-0543eef08544","Type":"ContainerStarted","Data":"137a1d56be560ddf89629d2c42daa28d5189faaaecbe38b0ae63cdaadff02e3d"} Nov 02 22:47:58.260057 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:47:58.259999 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5ljqt" podStartSLOduration=381.92624708 podStartE2EDuration="6m24.259979793s" totalImagesPullingTime="2.333732713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:58.259924532 +0000 UTC m=+36.822287871" watchObservedRunningTime="2026-11-02 22:47:58.259979793 +0000 UTC m=+36.822343131" Nov 02 22:48:00.152496 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:00.152464 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/0.log" Nov 02 22:48:00.152951 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:00.152511 2573 generic.go:356] "Generic (PLEG): container finished" podID="a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5" containerID="2ddcc5b8f7b8eab2c436ca265c11356d563acbd7f838a2efd3f85bc14e6dcd81" exitCode=255 Nov 02 22:48:00.152951 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:00.152587 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" event={"ID":"a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5","Type":"ContainerDied","Data":"2ddcc5b8f7b8eab2c436ca265c11356d563acbd7f838a2efd3f85bc14e6dcd81"} Nov 02 22:48:00.152951 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:00.152947 2573 scope.go:122] "RemoveContainer" containerID="2ddcc5b8f7b8eab2c436ca265c11356d563acbd7f838a2efd3f85bc14e6dcd81" Nov 02 22:48:01.159089 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.159044 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-464jz" event={"ID":"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c","Type":"ContainerStarted","Data":"7239b669767a2db3a54679623bc6fa6aaae096af8e98695ab41a369d27ebc3a3"} Nov 02 22:48:01.161490 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.161463 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/1.log" Nov 02 22:48:01.162023 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.162004 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/0.log" Nov 02 22:48:01.162145 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.162045 2573 generic.go:356] "Generic (PLEG): container finished" podID="a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5" containerID="d0392486b96f5150bbd960a42fc25be88c7131a360215caf09827617b78edc13" exitCode=255 Nov 02 22:48:01.162377 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.162349 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" event={"ID":"a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5","Type":"ContainerDied","Data":"d0392486b96f5150bbd960a42fc25be88c7131a360215caf09827617b78edc13"} Nov 02 22:48:01.162471 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.162416 2573 scope.go:122] "RemoveContainer" containerID="2ddcc5b8f7b8eab2c436ca265c11356d563acbd7f838a2efd3f85bc14e6dcd81" Nov 02 22:48:01.162471 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.162455 2573 scope.go:122] "RemoveContainer" containerID="d0392486b96f5150bbd960a42fc25be88c7131a360215caf09827617b78edc13" Nov 02 22:48:01.162685 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:01.162658 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-fj4n7_openshift-console-operator(a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" podUID="a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5" Nov 02 22:48:01.164385 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.164359 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xglvd" event={"ID":"071262de-534b-4d01-bc20-784db9852f69","Type":"ContainerStarted","Data":"4fd81f46795417b11a87ac4342bd040579866e3f757a8893efa1d247f9d07056"} Nov 02 22:48:01.166580 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.166551 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt" event={"ID":"ae96a9c1-10a6-4050-bca0-6a2e7859d3c3","Type":"ContainerStarted","Data":"ea853af35c2d3f487a8ee38b3a66286d0b5854ad7e11250bb21104980d781f30"} Nov 02 22:48:01.277182 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.277110 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-464jz" podStartSLOduration=381.850295989 podStartE2EDuration="6m27.277093242s" totalImagesPullingTime="5.426797253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:01.275211222 +0000 UTC m=+39.837574569" watchObservedRunningTime="2026-11-02 22:48:01.277093242 +0000 UTC m=+39.839456579" Nov 02 22:48:01.517485 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.516185 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt" podStartSLOduration=382.055050208 podStartE2EDuration="6m27.516165793s" totalImagesPullingTime="5.461115585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:01.514587857 +0000 UTC m=+40.076951219" watchObservedRunningTime="2026-11-02 22:48:01.516165793 +0000 UTC m=+40.078529131" Nov 02 22:48:01.630022 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.629957 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xglvd" podStartSLOduration=382.163048342 podStartE2EDuration="6m27.629941852s" totalImagesPullingTime="5.46689351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:01.628768254 +0000 UTC m=+40.191131594" watchObservedRunningTime="2026-11-02 22:48:01.629941852 +0000 UTC m=+40.192305186" Nov 02 22:48:01.668071 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.668039 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qlfjm"] Nov 02 22:48:01.691829 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.691798 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:01.730472 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.730400 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qlfjm"] Nov 02 22:48:01.748936 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.748904 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 22:48:01.749248 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.749230 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5z48j\"" Nov 02 22:48:01.759615 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.759586 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ee0440bc-1614-4209-8744-6d197a019b5c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:01.759756 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.759629 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ee0440bc-1614-4209-8744-6d197a019b5c-crio-socket\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:01.759756 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.759733 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ba063f-0db3-4f82-b280-40718c28b479-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qcz4m\" (UID: \"53ba063f-0db3-4f82-b280-40718c28b479\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qcz4m" Nov 02 22:48:01.759909 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.759882 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee0440bc-1614-4209-8744-6d197a019b5c-data-volume\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:01.760008 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.759935 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ee0440bc-1614-4209-8744-6d197a019b5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:01.760008 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:01.759950 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 22:48:01.760080 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:01.760028 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ba063f-0db3-4f82-b280-40718c28b479-cluster-monitoring-operator-tls podName:53ba063f-0db3-4f82-b280-40718c28b479 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:09.760007585 +0000 UTC m=+48.322370942 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/53ba063f-0db3-4f82-b280-40718c28b479-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qcz4m" (UID: "53ba063f-0db3-4f82-b280-40718c28b479") : secret "cluster-monitoring-operator-tls" not found Nov 02 22:48:01.760146 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.760099 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwnw9\" (UniqueName: \"kubernetes.io/projected/ee0440bc-1614-4209-8744-6d197a019b5c-kube-api-access-jwnw9\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:01.835017 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.834984 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 22:48:01.860550 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.860518 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee0440bc-1614-4209-8744-6d197a019b5c-data-volume\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:01.860731 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.860568 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ee0440bc-1614-4209-8744-6d197a019b5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:01.860731 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.860598 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dccc35b-6206-47f3-b901-bafc5f17b8ad-service-ca-bundle\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:48:01.860731 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.860641 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwnw9\" (UniqueName: \"kubernetes.io/projected/ee0440bc-1614-4209-8744-6d197a019b5c-kube-api-access-jwnw9\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:01.860731 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.860690 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-metrics-certs\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:48:01.860731 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:01.860714 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 22:48:01.860970 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:01.860763 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1dccc35b-6206-47f3-b901-bafc5f17b8ad-service-ca-bundle podName:1dccc35b-6206-47f3-b901-bafc5f17b8ad nodeName:}" failed. No retries permitted until 2026-11-02 22:48:09.860740303 +0000 UTC m=+48.423103619 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1dccc35b-6206-47f3-b901-bafc5f17b8ad-service-ca-bundle") pod "router-default-566bb4455-kq2j4" (UID: "1dccc35b-6206-47f3-b901-bafc5f17b8ad") : configmap references non-existent config key: service-ca.crt Nov 02 22:48:01.860970 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:01.860796 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee0440bc-1614-4209-8744-6d197a019b5c-insights-runtime-extractor-tls podName:ee0440bc-1614-4209-8744-6d197a019b5c nodeName:}" failed. No retries permitted until 2026-11-02 22:48:02.360784819 +0000 UTC m=+40.923148150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ee0440bc-1614-4209-8744-6d197a019b5c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qlfjm" (UID: "ee0440bc-1614-4209-8744-6d197a019b5c") : secret "insights-runtime-extractor-tls" not found Nov 02 22:48:01.860970 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:01.860817 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 22:48:01.860970 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.860850 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ee0440bc-1614-4209-8744-6d197a019b5c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:01.860970 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:01.860861 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-metrics-certs podName:1dccc35b-6206-47f3-b901-bafc5f17b8ad nodeName:}" failed. No retries permitted until 2026-11-02 22:48:09.860846608 +0000 UTC m=+48.423209944 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-metrics-certs") pod "router-default-566bb4455-kq2j4" (UID: "1dccc35b-6206-47f3-b901-bafc5f17b8ad") : secret "router-metrics-certs-default" not found Nov 02 22:48:01.860970 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.860862 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee0440bc-1614-4209-8744-6d197a019b5c-data-volume\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:01.860970 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:01.860892 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 22:48:01.860970 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.860907 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ee0440bc-1614-4209-8744-6d197a019b5c-crio-socket\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:01.860970 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:01.860927 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ee0440bc-1614-4209-8744-6d197a019b5c-kube-rbac-proxy-cm podName:ee0440bc-1614-4209-8744-6d197a019b5c nodeName:}" failed. No retries permitted until 2026-11-02 22:48:02.360914678 +0000 UTC m=+40.923277993 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ee0440bc-1614-4209-8744-6d197a019b5c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qlfjm" (UID: "ee0440bc-1614-4209-8744-6d197a019b5c") : configmap "kube-rbac-proxy" not found Nov 02 22:48:01.860970 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.860945 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ee0440bc-1614-4209-8744-6d197a019b5c-crio-socket\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:01.921260 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.921234 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwnw9\" (UniqueName: \"kubernetes.io/projected/ee0440bc-1614-4209-8744-6d197a019b5c-kube-api-access-jwnw9\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:01.962025 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.961998 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rf5x6\" (UID: \"2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rf5x6" Nov 02 22:48:01.962116 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.962058 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nj7zh\" (UID: \"d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nj7zh" Nov 02 22:48:01.962214 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:01.962175 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 22:48:01.962214 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:01.962180 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-tls\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:48:01.962301 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:01.962175 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 22:48:01.962301 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:01.962229 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 22:48:01.962301 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:01.962246 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bc8cbc8c-hghqr: secret "image-registry-tls" not found Nov 02 22:48:01.962301 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:01.962256 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-samples-operator-tls podName:2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc nodeName:}" failed. No retries permitted until 2026-11-02 22:48:09.962241615 +0000 UTC m=+48.524604949 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-rf5x6" (UID: "2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc") : secret "samples-operator-tls" not found Nov 02 22:48:01.962301 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:01.962276 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542-networking-console-plugin-cert podName:d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:09.962269614 +0000 UTC m=+48.524632928 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-nj7zh" (UID: "d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542") : secret "networking-console-plugin-cert" not found Nov 02 22:48:01.962301 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:01.962288 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-tls podName:2d578e40-60d8-4ebd-b0b5-2f5c274c7903 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:09.962282842 +0000 UTC m=+48.524646156 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-tls") pod "image-registry-bc8cbc8c-hghqr" (UID: "2d578e40-60d8-4ebd-b0b5-2f5c274c7903") : secret "image-registry-tls" not found Nov 02 22:48:02.000459 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:02.000430 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ttdqt"] Nov 02 22:48:02.018926 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:02.018906 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ttdqt" Nov 02 22:48:02.063040 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:02.062981 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-registry-tls\") pod \"image-registry-86798bc874-lbrfv\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:48:02.063224 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:02.063068 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72ef59a1-3734-47b7-9074-f683ca94dd32-metrics-tls\") pod \"dns-default-c8kg9\" (UID: \"72ef59a1-3734-47b7-9074-f683ca94dd32\") " pod="openshift-dns/dns-default-c8kg9" Nov 02 22:48:02.063224 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:02.063177 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 22:48:02.063224 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:02.063200 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86798bc874-lbrfv: secret "image-registry-tls" not found Nov 02 22:48:02.063224 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:02.063177 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8t9t\" (UniqueName: \"kubernetes.io/projected/06f6f89f-b8f9-45ec-b50a-f3407bfe7ef8-kube-api-access-b8t9t\") pod \"migrator-66df5d496b-ttdqt\" (UID: \"06f6f89f-b8f9-45ec-b50a-f3407bfe7ef8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ttdqt" Nov 02 22:48:02.063462 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:02.063241 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 22:48:02.063462 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:02.063316 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72ef59a1-3734-47b7-9074-f683ca94dd32-metrics-tls podName:72ef59a1-3734-47b7-9074-f683ca94dd32 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:10.063296162 +0000 UTC m=+48.625659497 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72ef59a1-3734-47b7-9074-f683ca94dd32-metrics-tls") pod "dns-default-c8kg9" (UID: "72ef59a1-3734-47b7-9074-f683ca94dd32") : secret "dns-default-metrics-tls" not found Nov 02 22:48:02.063462 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:02.063360 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-registry-tls podName:db94cbdb-79e6-4d1d-bf66-269fbab94abe nodeName:}" failed. No retries permitted until 2026-11-02 22:48:10.063344706 +0000 UTC m=+48.625708030 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-registry-tls") pod "image-registry-86798bc874-lbrfv" (UID: "db94cbdb-79e6-4d1d-bf66-269fbab94abe") : secret "image-registry-tls" not found Nov 02 22:48:02.063628 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:02.063505 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ttdqt"] Nov 02 22:48:02.073061 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:02.073039 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 22:48:02.073061 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:02.073055 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 22:48:02.073271 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:02.073062 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-cv7fc\"" Nov 02 22:48:02.164639 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:02.164607 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8t9t\" (UniqueName: \"kubernetes.io/projected/06f6f89f-b8f9-45ec-b50a-f3407bfe7ef8-kube-api-access-b8t9t\") pod \"migrator-66df5d496b-ttdqt\" (UID: \"06f6f89f-b8f9-45ec-b50a-f3407bfe7ef8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ttdqt" Nov 02 22:48:02.171499 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:02.171467 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2qpkq" event={"ID":"566b2dae-dd3e-4c63-b4d2-e30667a5fd07","Type":"ContainerStarted","Data":"40e7e0de9b0308d7dc71409b49bfc0a64d74e54faa7e1e44826fc617662a5dc8"} Nov 02 22:48:02.171665 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:02.171547 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2qpkq" Nov 02 22:48:02.173005 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:02.172981 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nbswd" event={"ID":"ff5ad4d1-4fd9-4374-b11c-78596c9cafa8","Type":"ContainerStarted","Data":"23590130b1b98215b84eba075266d019cc913034fe10b413daee05525cf2cbc5"} Nov 02 22:48:02.174814 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:02.174795 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/1.log" Nov 02 22:48:02.175094 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:02.175071 2573 scope.go:122] "RemoveContainer" containerID="d0392486b96f5150bbd960a42fc25be88c7131a360215caf09827617b78edc13" Nov 02 22:48:02.175295 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:02.175271 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-fj4n7_openshift-console-operator(a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" podUID="a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5" Nov 02 22:48:02.176417 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:02.176396 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6jd4b" event={"ID":"215a42f8-881d-4cc4-b670-ea9f4aefb80b","Type":"ContainerStarted","Data":"cb20a830fe5d204086058fc296f0f622ef94effccf18a694a747a10996a96f4c"} Nov 02 22:48:02.227441 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:02.227303 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8t9t\" (UniqueName: \"kubernetes.io/projected/06f6f89f-b8f9-45ec-b50a-f3407bfe7ef8-kube-api-access-b8t9t\") pod \"migrator-66df5d496b-ttdqt\" (UID: \"06f6f89f-b8f9-45ec-b50a-f3407bfe7ef8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ttdqt" Nov 02 22:48:02.286660 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:02.286558 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2qpkq" podStartSLOduration=33.931436604 podStartE2EDuration="40.286539401s" totalImagesPullingTime="6.355102797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:02.28579425 +0000 UTC m=+40.848157587" watchObservedRunningTime="2026-11-02 22:48:02.286539401 +0000 UTC m=+40.848902739" Nov 02 22:48:02.340735 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:02.340710 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ttdqt" Nov 02 22:48:02.366745 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:02.366716 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ee0440bc-1614-4209-8744-6d197a019b5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:02.366869 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:02.366853 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ee0440bc-1614-4209-8744-6d197a019b5c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:02.366944 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:02.366864 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 22:48:02.366944 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:02.366928 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee0440bc-1614-4209-8744-6d197a019b5c-insights-runtime-extractor-tls podName:ee0440bc-1614-4209-8744-6d197a019b5c nodeName:}" failed. No retries permitted until 2026-11-02 22:48:03.366910114 +0000 UTC m=+41.929273446 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ee0440bc-1614-4209-8744-6d197a019b5c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qlfjm" (UID: "ee0440bc-1614-4209-8744-6d197a019b5c") : secret "insights-runtime-extractor-tls" not found Nov 02 22:48:02.366944 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:02.366930 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 22:48:02.367109 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:02.366965 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ee0440bc-1614-4209-8744-6d197a019b5c-kube-rbac-proxy-cm podName:ee0440bc-1614-4209-8744-6d197a019b5c nodeName:}" failed. No retries permitted until 2026-11-02 22:48:03.366955004 +0000 UTC m=+41.929318320 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ee0440bc-1614-4209-8744-6d197a019b5c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qlfjm" (UID: "ee0440bc-1614-4209-8744-6d197a019b5c") : configmap "kube-rbac-proxy" not found Nov 02 22:48:02.400924 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:02.400865 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6jd4b" podStartSLOduration=3.014735799 podStartE2EDuration="9.400851592s" totalImagesPullingTime="6.386115793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:47:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:02.399864825 +0000 UTC m=+40.962228163" watchObservedRunningTime="2026-11-02 22:48:02.400851592 +0000 UTC m=+40.963214930" Nov 02 22:48:02.635094 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:02.635030 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nbswd" podStartSLOduration=377.611459738 podStartE2EDuration="6m24.635011218s" totalImagesPullingTime="7.02355148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:41:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:02.634409971 +0000 UTC m=+41.196773309" watchObservedRunningTime="2026-11-02 22:48:02.635011218 +0000 UTC m=+41.197374558" Nov 02 22:48:02.726723 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:02.726697 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ttdqt"] Nov 02 22:48:02.729281 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:48:02.729252 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f6f89f_b8f9_45ec_b50a_f3407bfe7ef8.slice/crio-9ca958337e09414cec8117b907118ca7d199b2255b08746ec08bc6182ccdeebd WatchSource:0}: Error finding container 9ca958337e09414cec8117b907118ca7d199b2255b08746ec08bc6182ccdeebd: Status 404 returned error can't find the container with id 9ca958337e09414cec8117b907118ca7d199b2255b08746ec08bc6182ccdeebd Nov 02 22:48:03.181447 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:03.181404 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ttdqt" event={"ID":"06f6f89f-b8f9-45ec-b50a-f3407bfe7ef8","Type":"ContainerStarted","Data":"9ca958337e09414cec8117b907118ca7d199b2255b08746ec08bc6182ccdeebd"} Nov 02 22:48:03.375939 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:03.375900 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ee0440bc-1614-4209-8744-6d197a019b5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:03.376105 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:03.376017 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ee0440bc-1614-4209-8744-6d197a019b5c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:03.376105 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:03.376042 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 22:48:03.376105 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:03.376084 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 22:48:03.376105 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:03.376102 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee0440bc-1614-4209-8744-6d197a019b5c-insights-runtime-extractor-tls podName:ee0440bc-1614-4209-8744-6d197a019b5c nodeName:}" failed. No retries permitted until 2026-11-02 22:48:05.376088004 +0000 UTC m=+43.938451337 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ee0440bc-1614-4209-8744-6d197a019b5c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qlfjm" (UID: "ee0440bc-1614-4209-8744-6d197a019b5c") : secret "insights-runtime-extractor-tls" not found Nov 02 22:48:03.376343 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:03.376140 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ee0440bc-1614-4209-8744-6d197a019b5c-kube-rbac-proxy-cm podName:ee0440bc-1614-4209-8744-6d197a019b5c nodeName:}" failed. No retries permitted until 2026-11-02 22:48:05.376113723 +0000 UTC m=+43.938477057 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ee0440bc-1614-4209-8744-6d197a019b5c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qlfjm" (UID: "ee0440bc-1614-4209-8744-6d197a019b5c") : configmap "kube-rbac-proxy" not found Nov 02 22:48:04.332657 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:04.332623 2573 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" Nov 02 22:48:04.333109 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:04.332687 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" Nov 02 22:48:04.333200 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:04.333173 2573 scope.go:122] "RemoveContainer" containerID="d0392486b96f5150bbd960a42fc25be88c7131a360215caf09827617b78edc13" Nov 02 22:48:04.333402 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:04.333380 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-fj4n7_openshift-console-operator(a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" podUID="a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5" Nov 02 22:48:05.189325 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:05.189241 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ttdqt" event={"ID":"06f6f89f-b8f9-45ec-b50a-f3407bfe7ef8","Type":"ContainerStarted","Data":"68d65c617a0965b7226c6716b12cfbc021dc03fb152d03c84dcb7bf37e17e8cd"} Nov 02 22:48:05.303818 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:05.303769 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ttdqt" podStartSLOduration=2.8085363169999997 podStartE2EDuration="4.303755693s" totalImagesPullingTime="1.495219376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:48:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:05.302869472 +0000 UTC m=+43.865232825" watchObservedRunningTime="2026-11-02 22:48:05.303755693 +0000 UTC m=+43.866119030" Nov 02 22:48:05.394575 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:05.394544 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ee0440bc-1614-4209-8744-6d197a019b5c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:05.395012 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:05.394629 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ee0440bc-1614-4209-8744-6d197a019b5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:05.395012 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:05.394732 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 22:48:05.395012 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:05.394795 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee0440bc-1614-4209-8744-6d197a019b5c-insights-runtime-extractor-tls podName:ee0440bc-1614-4209-8744-6d197a019b5c nodeName:}" failed. No retries permitted until 2026-11-02 22:48:09.394782823 +0000 UTC m=+47.957146139 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ee0440bc-1614-4209-8744-6d197a019b5c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qlfjm" (UID: "ee0440bc-1614-4209-8744-6d197a019b5c") : secret "insights-runtime-extractor-tls" not found Nov 02 22:48:05.395184 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:05.395097 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ee0440bc-1614-4209-8744-6d197a019b5c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:09.431578 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:09.431542 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ee0440bc-1614-4209-8744-6d197a019b5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:09.432034 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:09.431706 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 22:48:09.432034 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:09.431797 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee0440bc-1614-4209-8744-6d197a019b5c-insights-runtime-extractor-tls podName:ee0440bc-1614-4209-8744-6d197a019b5c nodeName:}" failed. No retries permitted until 2026-11-02 22:48:17.431781624 +0000 UTC m=+55.994144939 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ee0440bc-1614-4209-8744-6d197a019b5c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qlfjm" (UID: "ee0440bc-1614-4209-8744-6d197a019b5c") : secret "insights-runtime-extractor-tls" not found Nov 02 22:48:09.835767 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:09.835729 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ba063f-0db3-4f82-b280-40718c28b479-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qcz4m\" (UID: \"53ba063f-0db3-4f82-b280-40718c28b479\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qcz4m" Nov 02 22:48:09.835895 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:09.835852 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 22:48:09.835949 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:09.835911 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ba063f-0db3-4f82-b280-40718c28b479-cluster-monitoring-operator-tls podName:53ba063f-0db3-4f82-b280-40718c28b479 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:25.83589743 +0000 UTC m=+64.398260745 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/53ba063f-0db3-4f82-b280-40718c28b479-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qcz4m" (UID: "53ba063f-0db3-4f82-b280-40718c28b479") : secret "cluster-monitoring-operator-tls" not found Nov 02 22:48:09.936769 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:09.936739 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dccc35b-6206-47f3-b901-bafc5f17b8ad-service-ca-bundle\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:48:09.936934 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:09.936795 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-metrics-certs\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:48:09.936934 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:09.936924 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1dccc35b-6206-47f3-b901-bafc5f17b8ad-service-ca-bundle podName:1dccc35b-6206-47f3-b901-bafc5f17b8ad nodeName:}" failed. No retries permitted until 2026-11-02 22:48:25.936905234 +0000 UTC m=+64.499268568 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1dccc35b-6206-47f3-b901-bafc5f17b8ad-service-ca-bundle") pod "router-default-566bb4455-kq2j4" (UID: "1dccc35b-6206-47f3-b901-bafc5f17b8ad") : configmap references non-existent config key: service-ca.crt Nov 02 22:48:09.936934 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:09.936925 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 22:48:09.937050 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:09.936971 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-metrics-certs podName:1dccc35b-6206-47f3-b901-bafc5f17b8ad nodeName:}" failed. No retries permitted until 2026-11-02 22:48:25.936961702 +0000 UTC m=+64.499325032 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-metrics-certs") pod "router-default-566bb4455-kq2j4" (UID: "1dccc35b-6206-47f3-b901-bafc5f17b8ad") : secret "router-metrics-certs-default" not found Nov 02 22:48:10.038281 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:10.038244 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rf5x6\" (UID: \"2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rf5x6" Nov 02 22:48:10.038449 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:10.038289 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nj7zh\" (UID: \"d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nj7zh" Nov 02 22:48:10.038449 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:10.038353 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-tls\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:48:10.038449 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:10.038392 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 22:48:10.038449 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:10.038446 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-samples-operator-tls podName:2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc nodeName:}" failed. No retries permitted until 2026-11-02 22:48:26.038432466 +0000 UTC m=+64.600795781 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-rf5x6" (UID: "2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc") : secret "samples-operator-tls" not found Nov 02 22:48:10.038595 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:10.038448 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 22:48:10.038595 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:10.038464 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bc8cbc8c-hghqr: secret "image-registry-tls" not found Nov 02 22:48:10.038595 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:10.038502 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 22:48:10.038595 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:10.038519 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-tls podName:2d578e40-60d8-4ebd-b0b5-2f5c274c7903 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:26.038502356 +0000 UTC m=+64.600865691 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-tls") pod "image-registry-bc8cbc8c-hghqr" (UID: "2d578e40-60d8-4ebd-b0b5-2f5c274c7903") : secret "image-registry-tls" not found Nov 02 22:48:10.038595 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:10.038541 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542-networking-console-plugin-cert podName:d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:26.038529651 +0000 UTC m=+64.600892966 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-nj7zh" (UID: "d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542") : secret "networking-console-plugin-cert" not found Nov 02 22:48:10.139717 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:10.139653 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-registry-tls\") pod \"image-registry-86798bc874-lbrfv\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:48:10.139717 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:10.139714 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72ef59a1-3734-47b7-9074-f683ca94dd32-metrics-tls\") pod \"dns-default-c8kg9\" (UID: \"72ef59a1-3734-47b7-9074-f683ca94dd32\") " pod="openshift-dns/dns-default-c8kg9" Nov 02 22:48:10.139840 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:10.139801 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 22:48:10.139840 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:10.139821 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86798bc874-lbrfv: secret "image-registry-tls" not found Nov 02 22:48:10.139900 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:10.139858 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 22:48:10.139900 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:10.139874 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-registry-tls podName:db94cbdb-79e6-4d1d-bf66-269fbab94abe nodeName:}" failed. No retries permitted until 2026-11-02 22:48:26.139857245 +0000 UTC m=+64.702220585 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-registry-tls") pod "image-registry-86798bc874-lbrfv" (UID: "db94cbdb-79e6-4d1d-bf66-269fbab94abe") : secret "image-registry-tls" not found Nov 02 22:48:10.139900 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:10.139899 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72ef59a1-3734-47b7-9074-f683ca94dd32-metrics-tls podName:72ef59a1-3734-47b7-9074-f683ca94dd32 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:26.139887665 +0000 UTC m=+64.702250980 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72ef59a1-3734-47b7-9074-f683ca94dd32-metrics-tls") pod "dns-default-c8kg9" (UID: "72ef59a1-3734-47b7-9074-f683ca94dd32") : secret "dns-default-metrics-tls" not found Nov 02 22:48:12.392013 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:12.391981 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lhbwd_cb204d6b-3758-4db4-bd68-0eacec87b805/dns-node-resolver/0.log" Nov 02 22:48:12.918423 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:12.918397 2573 scope.go:122] "RemoveContainer" containerID="d0392486b96f5150bbd960a42fc25be88c7131a360215caf09827617b78edc13" Nov 02 22:48:13.116637 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:13.116588 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mp88w_fc560bbe-b31a-48e8-8416-dda72f49dae1/node-ca/0.log" Nov 02 22:48:13.215026 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:13.214958 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 22:48:13.215353 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:13.215337 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/1.log" Nov 02 22:48:13.215422 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:13.215371 2573 generic.go:356] "Generic (PLEG): container finished" podID="a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5" containerID="de773a7c0efbc6b4377ddff49679469091bfed1e165b3ad6bfdfa9bd6760db62" exitCode=255 Nov 02 22:48:13.215459 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:13.215425 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" event={"ID":"a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5","Type":"ContainerDied","Data":"de773a7c0efbc6b4377ddff49679469091bfed1e165b3ad6bfdfa9bd6760db62"} Nov 02 22:48:13.215493 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:13.215468 2573 scope.go:122] "RemoveContainer" containerID="d0392486b96f5150bbd960a42fc25be88c7131a360215caf09827617b78edc13" Nov 02 22:48:13.215777 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:13.215751 2573 scope.go:122] "RemoveContainer" containerID="de773a7c0efbc6b4377ddff49679469091bfed1e165b3ad6bfdfa9bd6760db62" Nov 02 22:48:13.215966 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:13.215948 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-fj4n7_openshift-console-operator(a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" podUID="a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5" Nov 02 22:48:13.716091 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:13.716059 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6jd4b_215a42f8-881d-4cc4-b670-ea9f4aefb80b/serve-healthcheck-canary/0.log" Nov 02 22:48:14.219983 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:14.219951 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 22:48:14.315952 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:14.315925 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-ttdqt_06f6f89f-b8f9-45ec-b50a-f3407bfe7ef8/migrator/0.log" Nov 02 22:48:14.332869 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:14.332850 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" Nov 02 22:48:14.332947 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:14.332889 2573 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" Nov 02 22:48:14.333197 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:14.333186 2573 scope.go:122] "RemoveContainer" containerID="de773a7c0efbc6b4377ddff49679469091bfed1e165b3ad6bfdfa9bd6760db62" Nov 02 22:48:14.333381 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:14.333364 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-fj4n7_openshift-console-operator(a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" podUID="a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5" Nov 02 22:48:14.517173 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:14.517083 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt_ae96a9c1-10a6-4050-bca0-6a2e7859d3c3/kube-storage-version-migrator-operator/0.log" Nov 02 22:48:17.103043 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:17.103015 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:17.507431 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:17.507340 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ee0440bc-1614-4209-8744-6d197a019b5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:17.509936 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:17.509905 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ee0440bc-1614-4209-8744-6d197a019b5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:17.655356 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:17.655321 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:18.026071 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:18.026039 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qlfjm"] Nov 02 22:48:18.059060 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:48:18.059020 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee0440bc_1614_4209_8744_6d197a019b5c.slice/crio-31b84ffae7960055d9afb0250f2959a11bbbc6bd0d76c87a2c5a2c9021b420e0 WatchSource:0}: Error finding container 31b84ffae7960055d9afb0250f2959a11bbbc6bd0d76c87a2c5a2c9021b420e0: Status 404 returned error can't find the container with id 31b84ffae7960055d9afb0250f2959a11bbbc6bd0d76c87a2c5a2c9021b420e0 Nov 02 22:48:18.234366 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:18.234334 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qlfjm" event={"ID":"ee0440bc-1614-4209-8744-6d197a019b5c","Type":"ContainerStarted","Data":"d0aab0ff1a37b96d568cc3fc9e1db815ef2c273d8e141c2e7c2d5c8b766cfdd7"} Nov 02 22:48:18.234711 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:18.234390 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qlfjm" event={"ID":"ee0440bc-1614-4209-8744-6d197a019b5c","Type":"ContainerStarted","Data":"31b84ffae7960055d9afb0250f2959a11bbbc6bd0d76c87a2c5a2c9021b420e0"} Nov 02 22:48:20.245978 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:20.245945 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qlfjm" event={"ID":"ee0440bc-1614-4209-8744-6d197a019b5c","Type":"ContainerStarted","Data":"c004dcee6412c3ebea528b581dbc94b850c8c6743355280a4a7c1c02ea6703cd"} Nov 02 22:48:22.254882 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:22.254839 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qlfjm" event={"ID":"ee0440bc-1614-4209-8744-6d197a019b5c","Type":"ContainerStarted","Data":"23acd917e9e3e24792b17d81830d5e343f925fa3c07169f52bfd68e77a20b4f7"} Nov 02 22:48:22.422364 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:22.422308 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qlfjm" podStartSLOduration=18.237846996 podStartE2EDuration="21.42229388s" totalImagesPullingTime="3.184446884s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:48:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:22.371749506 +0000 UTC m=+60.934112845" watchObservedRunningTime="2026-11-02 22:48:22.42229388 +0000 UTC m=+60.984657217" Nov 02 22:48:25.882036 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:25.882002 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ba063f-0db3-4f82-b280-40718c28b479-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qcz4m\" (UID: \"53ba063f-0db3-4f82-b280-40718c28b479\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qcz4m" Nov 02 22:48:25.884592 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:25.884569 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ba063f-0db3-4f82-b280-40718c28b479-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qcz4m\" (UID: \"53ba063f-0db3-4f82-b280-40718c28b479\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qcz4m" Nov 02 22:48:25.982870 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:25.982835 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dccc35b-6206-47f3-b901-bafc5f17b8ad-service-ca-bundle\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:48:25.983053 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:25.982900 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-metrics-certs\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:48:25.983478 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:25.983452 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dccc35b-6206-47f3-b901-bafc5f17b8ad-service-ca-bundle\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:48:25.985353 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:25.985332 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-metrics-certs\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:48:26.070047 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.070014 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qcz4m" Nov 02 22:48:26.084225 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.084198 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rf5x6\" (UID: \"2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rf5x6" Nov 02 22:48:26.084319 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.084240 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nj7zh\" (UID: \"d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nj7zh" Nov 02 22:48:26.084319 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.084293 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-tls\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:48:26.086856 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.086834 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-tls\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:48:26.086968 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.086945 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nj7zh\" (UID: \"d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nj7zh" Nov 02 22:48:26.087280 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.087261 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rf5x6\" (UID: \"2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rf5x6" Nov 02 22:48:26.109335 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.109303 2573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-86798bc874-lbrfv"] Nov 02 22:48:26.109567 ip-10-0-135-176 kubenswrapper[2573]: E1102 22:48:26.109544 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-86798bc874-lbrfv" podUID="db94cbdb-79e6-4d1d-bf66-269fbab94abe" Nov 02 22:48:26.176601 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.176542 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:48:26.184805 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.184781 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-registry-tls\") pod \"image-registry-86798bc874-lbrfv\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:48:26.184911 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.184823 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72ef59a1-3734-47b7-9074-f683ca94dd32-metrics-tls\") pod \"dns-default-c8kg9\" (UID: \"72ef59a1-3734-47b7-9074-f683ca94dd32\") " pod="openshift-dns/dns-default-c8kg9" Nov 02 22:48:26.187382 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.187358 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-registry-tls\") pod \"image-registry-86798bc874-lbrfv\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:48:26.187564 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.187546 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72ef59a1-3734-47b7-9074-f683ca94dd32-metrics-tls\") pod \"dns-default-c8kg9\" (UID: \"72ef59a1-3734-47b7-9074-f683ca94dd32\") " pod="openshift-dns/dns-default-c8kg9" Nov 02 22:48:26.247579 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.247555 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rf5x6" Nov 02 22:48:26.268728 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.268706 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:48:26.276281 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.276261 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:48:26.316421 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.316378 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-rncv6\"" Nov 02 22:48:26.326144 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.326108 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-nj7zh" Nov 02 22:48:26.346442 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.346423 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9g8qt\"" Nov 02 22:48:26.356550 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.356531 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:48:26.386511 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.386484 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db94cbdb-79e6-4d1d-bf66-269fbab94abe-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db94cbdb-79e6-4d1d-bf66-269fbab94abe-trusted-ca\") pod \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " Nov 02 22:48:26.386643 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.386539 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-registry-tls\" (UniqueName: \"kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-registry-tls\") pod \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " Nov 02 22:48:26.386643 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.386558 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-bound-sa-token\") pod \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " Nov 02 22:48:26.386643 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.386576 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db94cbdb-79e6-4d1d-bf66-269fbab94abe-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/db94cbdb-79e6-4d1d-bf66-269fbab94abe-image-registry-private-configuration\") pod \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " Nov 02 22:48:26.386643 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.386602 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db94cbdb-79e6-4d1d-bf66-269fbab94abe-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/db94cbdb-79e6-4d1d-bf66-269fbab94abe-ca-trust-extracted\") pod \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " Nov 02 22:48:26.386800 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.386715 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db94cbdb-79e6-4d1d-bf66-269fbab94abe-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/db94cbdb-79e6-4d1d-bf66-269fbab94abe-installation-pull-secrets\") pod \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " Nov 02 22:48:26.386800 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.386742 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db94cbdb-79e6-4d1d-bf66-269fbab94abe-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/db94cbdb-79e6-4d1d-bf66-269fbab94abe-registry-certificates\") pod \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " Nov 02 22:48:26.386800 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.386763 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-kube-api-access-m4d65\" (UniqueName: \"kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-kube-api-access-m4d65\") pod \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\" (UID: \"db94cbdb-79e6-4d1d-bf66-269fbab94abe\") " Nov 02 22:48:26.387223 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.387195 2573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db94cbdb-79e6-4d1d-bf66-269fbab94abe-trusted-ca" pod "db94cbdb-79e6-4d1d-bf66-269fbab94abe" (UID: "db94cbdb-79e6-4d1d-bf66-269fbab94abe"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:48:26.388105 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.388067 2573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db94cbdb-79e6-4d1d-bf66-269fbab94abe-registry-certificates" pod "db94cbdb-79e6-4d1d-bf66-269fbab94abe" (UID: "db94cbdb-79e6-4d1d-bf66-269fbab94abe"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:48:26.388264 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.388233 2573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db94cbdb-79e6-4d1d-bf66-269fbab94abe-ca-trust-extracted" pod "db94cbdb-79e6-4d1d-bf66-269fbab94abe" (UID: "db94cbdb-79e6-4d1d-bf66-269fbab94abe"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:48:26.393119 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.393095 2573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db94cbdb-79e6-4d1d-bf66-269fbab94abe-image-registry-private-configuration" pod "db94cbdb-79e6-4d1d-bf66-269fbab94abe" (UID: "db94cbdb-79e6-4d1d-bf66-269fbab94abe"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:48:26.393239 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.393172 2573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-bound-sa-token" pod "db94cbdb-79e6-4d1d-bf66-269fbab94abe" (UID: "db94cbdb-79e6-4d1d-bf66-269fbab94abe"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:48:26.393239 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.393186 2573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-registry-tls" pod "db94cbdb-79e6-4d1d-bf66-269fbab94abe" (UID: "db94cbdb-79e6-4d1d-bf66-269fbab94abe"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:48:26.393323 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.393247 2573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db94cbdb-79e6-4d1d-bf66-269fbab94abe-installation-pull-secrets" pod "db94cbdb-79e6-4d1d-bf66-269fbab94abe" (UID: "db94cbdb-79e6-4d1d-bf66-269fbab94abe"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:48:26.393323 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.393256 2573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-kube-api-access-m4d65" pod "db94cbdb-79e6-4d1d-bf66-269fbab94abe" (UID: "db94cbdb-79e6-4d1d-bf66-269fbab94abe"). InnerVolumeSpecName "kube-api-access-m4d65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:48:26.441935 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.441877 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-qcz4m"] Nov 02 22:48:26.461963 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.461938 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qf97t\"" Nov 02 22:48:26.471806 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.471787 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c8kg9" Nov 02 22:48:26.482432 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:48:26.482406 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ba063f_0db3_4f82_b280_40718c28b479.slice/crio-f4ef14f278f6ff49286a2be511bd4cd87dff2ceacccefce6ff649844e972b274 WatchSource:0}: Error finding container f4ef14f278f6ff49286a2be511bd4cd87dff2ceacccefce6ff649844e972b274: Status 404 returned error can't find the container with id f4ef14f278f6ff49286a2be511bd4cd87dff2ceacccefce6ff649844e972b274 Nov 02 22:48:26.487751 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.487644 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db94cbdb-79e6-4d1d-bf66-269fbab94abe-trusted-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:48:26.487751 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.487667 2573 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-registry-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:48:26.487751 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.487683 2573 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-bound-sa-token\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:48:26.487751 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.487699 2573 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/db94cbdb-79e6-4d1d-bf66-269fbab94abe-image-registry-private-configuration\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:48:26.487751 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.487710 2573 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/db94cbdb-79e6-4d1d-bf66-269fbab94abe-ca-trust-extracted\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:48:26.487751 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.487719 2573 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/db94cbdb-79e6-4d1d-bf66-269fbab94abe-installation-pull-secrets\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:48:26.487751 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.487728 2573 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/db94cbdb-79e6-4d1d-bf66-269fbab94abe-registry-certificates\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:48:26.487751 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.487737 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4d65\" (UniqueName: \"kubernetes.io/projected/db94cbdb-79e6-4d1d-bf66-269fbab94abe-kube-api-access-m4d65\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:48:26.547604 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.547582 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-566bb4455-kq2j4"] Nov 02 22:48:26.549892 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:48:26.549863 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dccc35b_6206_47f3_b901_bafc5f17b8ad.slice/crio-690e42623d04fa5aa3d49b1cbe7f4ed44923da61e38c096bdb85559d8705aed1 WatchSource:0}: Error finding container 690e42623d04fa5aa3d49b1cbe7f4ed44923da61e38c096bdb85559d8705aed1: Status 404 returned error can't find the container with id 690e42623d04fa5aa3d49b1cbe7f4ed44923da61e38c096bdb85559d8705aed1 Nov 02 22:48:26.620554 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.620532 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rf5x6"] Nov 02 22:48:26.692890 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.692865 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-nj7zh"] Nov 02 22:48:26.695312 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:48:26.695285 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9fd0cb2_d80d_4287_b9ca_d8b1d3d75542.slice/crio-2035e8f997a9e7d0935f37f90194f850ae3c24b21829a8b22471026bfae92ce8 WatchSource:0}: Error finding container 2035e8f997a9e7d0935f37f90194f850ae3c24b21829a8b22471026bfae92ce8: Status 404 returned error can't find the container with id 2035e8f997a9e7d0935f37f90194f850ae3c24b21829a8b22471026bfae92ce8 Nov 02 22:48:26.736494 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.736465 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bc8cbc8c-hghqr"] Nov 02 22:48:26.739074 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:48:26.739048 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d578e40_60d8_4ebd_b0b5_2f5c274c7903.slice/crio-4fba275614bd2ddaf44e9966d406d568f3b834031d711e5499040da2ca9c527d WatchSource:0}: Error finding container 4fba275614bd2ddaf44e9966d406d568f3b834031d711e5499040da2ca9c527d: Status 404 returned error can't find the container with id 4fba275614bd2ddaf44e9966d406d568f3b834031d711e5499040da2ca9c527d Nov 02 22:48:26.847165 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.847118 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c8kg9"] Nov 02 22:48:26.850027 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:48:26.849996 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72ef59a1_3734_47b7_9074_f683ca94dd32.slice/crio-2393b59785260194d19a54699f33e597481d2bf7ebd4c3b1679768477af03b4e WatchSource:0}: Error finding container 2393b59785260194d19a54699f33e597481d2bf7ebd4c3b1679768477af03b4e: Status 404 returned error can't find the container with id 2393b59785260194d19a54699f33e597481d2bf7ebd4c3b1679768477af03b4e Nov 02 22:48:26.890534 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.890509 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/417d26e1-2a9d-41e3-8940-69626662a2f3-original-pull-secret\") pod \"global-pull-secret-syncer-wbxxp\" (UID: \"417d26e1-2a9d-41e3-8940-69626662a2f3\") " pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:48:26.890941 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.890599 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82b3b5e2-09f1-49c0-8d72-c85187918223-metrics-certs\") pod \"network-metrics-daemon-465bf\" (UID: \"82b3b5e2-09f1-49c0-8d72-c85187918223\") " pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:48:26.941845 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.941812 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 22:48:26.941982 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.941866 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 22:48:26.953242 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.953182 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82b3b5e2-09f1-49c0-8d72-c85187918223-metrics-certs\") pod \"network-metrics-daemon-465bf\" (UID: \"82b3b5e2-09f1-49c0-8d72-c85187918223\") " pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:48:26.953242 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:26.953188 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/417d26e1-2a9d-41e3-8940-69626662a2f3-original-pull-secret\") pod \"global-pull-secret-syncer-wbxxp\" (UID: \"417d26e1-2a9d-41e3-8940-69626662a2f3\") " pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:48:27.108012 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:27.107976 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r8wwr\"" Nov 02 22:48:27.116331 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:27.116306 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zfnv8\"" Nov 02 22:48:27.117275 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:27.117252 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:48:27.126090 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:27.126074 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:48:27.273826 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:27.273720 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" event={"ID":"2d578e40-60d8-4ebd-b0b5-2f5c274c7903","Type":"ContainerStarted","Data":"7d50faaaa721bb4fbadb50f25d90ff8b0b97daecec94622ee6a6c11a4fcc8ab6"} Nov 02 22:48:27.273826 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:27.273776 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" event={"ID":"2d578e40-60d8-4ebd-b0b5-2f5c274c7903","Type":"ContainerStarted","Data":"4fba275614bd2ddaf44e9966d406d568f3b834031d711e5499040da2ca9c527d"} Nov 02 22:48:27.274033 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:27.273838 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:48:27.274981 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:27.274952 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-nj7zh" event={"ID":"d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542","Type":"ContainerStarted","Data":"2035e8f997a9e7d0935f37f90194f850ae3c24b21829a8b22471026bfae92ce8"} Nov 02 22:48:27.276086 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:27.276060 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rf5x6" event={"ID":"2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc","Type":"ContainerStarted","Data":"2762b05b8165a3da012d5281faadff367ecfd35437f6345d467e82f78916402a"} Nov 02 22:48:27.277447 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:27.277420 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c8kg9" event={"ID":"72ef59a1-3734-47b7-9074-f683ca94dd32","Type":"ContainerStarted","Data":"2393b59785260194d19a54699f33e597481d2bf7ebd4c3b1679768477af03b4e"} Nov 02 22:48:27.278821 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:27.278803 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-566bb4455-kq2j4" event={"ID":"1dccc35b-6206-47f3-b901-bafc5f17b8ad","Type":"ContainerStarted","Data":"37103159fba4692c0f7b2ed98bc2fe058ae5f2e5d005e8d15a1c38180454a757"} Nov 02 22:48:27.278906 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:27.278838 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-566bb4455-kq2j4" event={"ID":"1dccc35b-6206-47f3-b901-bafc5f17b8ad","Type":"ContainerStarted","Data":"690e42623d04fa5aa3d49b1cbe7f4ed44923da61e38c096bdb85559d8705aed1"} Nov 02 22:48:27.280014 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:27.279997 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86798bc874-lbrfv" Nov 02 22:48:27.280106 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:27.280031 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qcz4m" event={"ID":"53ba063f-0db3-4f82-b280-40718c28b479","Type":"ContainerStarted","Data":"f4ef14f278f6ff49286a2be511bd4cd87dff2ceacccefce6ff649844e972b274"} Nov 02 22:48:27.395806 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:27.395754 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" podStartSLOduration=56.395739811 podStartE2EDuration="56.395739811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:27.39433324 +0000 UTC m=+65.956696577" watchObservedRunningTime="2026-11-02 22:48:27.395739811 +0000 UTC m=+65.958103144" Nov 02 22:48:27.492385 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:27.492354 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-465bf"] Nov 02 22:48:27.495311 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:48:27.495281 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b3b5e2_09f1_49c0_8d72_c85187918223.slice/crio-e668ee624520a4afd64de9149efe141650aa4ccf915b34e76448afef488ed842 WatchSource:0}: Error finding container e668ee624520a4afd64de9149efe141650aa4ccf915b34e76448afef488ed842: Status 404 returned error can't find the container with id e668ee624520a4afd64de9149efe141650aa4ccf915b34e76448afef488ed842 Nov 02 22:48:27.514693 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:27.514648 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/router-default-566bb4455-kq2j4" podStartSLOduration=413.514635257 podStartE2EDuration="6m53.514635257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:27.513228175 +0000 UTC m=+66.075591512" watchObservedRunningTime="2026-11-02 22:48:27.514635257 +0000 UTC m=+66.076998595" Nov 02 22:48:27.541190 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:27.541170 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wbxxp"] Nov 02 22:48:27.543545 ip-10-0-135-176 kubenswrapper[2573]: W1102 22:48:27.543513 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417d26e1_2a9d_41e3_8940_69626662a2f3.slice/crio-061fd09ec7499698a85f2e9bb065f4baa4da0ef59ed108bd060b11ee0c4b7d9e WatchSource:0}: Error finding container 061fd09ec7499698a85f2e9bb065f4baa4da0ef59ed108bd060b11ee0c4b7d9e: Status 404 returned error can't find the container with id 061fd09ec7499698a85f2e9bb065f4baa4da0ef59ed108bd060b11ee0c4b7d9e Nov 02 22:48:27.690358 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:27.690324 2573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-86798bc874-lbrfv"] Nov 02 22:48:27.694505 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:27.694477 2573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-86798bc874-lbrfv"] Nov 02 22:48:27.925840 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:27.925803 2573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db94cbdb-79e6-4d1d-bf66-269fbab94abe" path="/var/lib/kubelet/pods/db94cbdb-79e6-4d1d-bf66-269fbab94abe/volumes" Nov 02 22:48:28.177962 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:28.177899 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:48:28.180782 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:28.180755 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:48:28.287980 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:28.287936 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-465bf" event={"ID":"82b3b5e2-09f1-49c0-8d72-c85187918223","Type":"ContainerStarted","Data":"e668ee624520a4afd64de9149efe141650aa4ccf915b34e76448afef488ed842"} Nov 02 22:48:28.290262 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:28.290191 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wbxxp" event={"ID":"417d26e1-2a9d-41e3-8940-69626662a2f3","Type":"ContainerStarted","Data":"061fd09ec7499698a85f2e9bb065f4baa4da0ef59ed108bd060b11ee0c4b7d9e"} Nov 02 22:48:28.291053 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:28.291006 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:48:28.292502 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:28.292485 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:48:29.298357 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:29.298320 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-nj7zh" event={"ID":"d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542","Type":"ContainerStarted","Data":"af0aa848762e50ff7ede2871c2a5522b8c1776802abbe1f5ec3e6444e3772e99"} Nov 02 22:48:29.417169 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:29.416289 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-86c478bb88-nj7zh" podStartSLOduration=376.464693706 podStartE2EDuration="6m18.416268954s" totalImagesPullingTime="1.951575248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:42:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:29.414453134 +0000 UTC m=+67.976816472" watchObservedRunningTime="2026-11-02 22:48:29.416268954 +0000 UTC m=+67.978632293" Nov 02 22:48:30.302724 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:30.302691 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-465bf" event={"ID":"82b3b5e2-09f1-49c0-8d72-c85187918223","Type":"ContainerStarted","Data":"0b4d4c2a800ebb22d595fa657362eac4b4214b5c3196c64fb401a05518ce2091"} Nov 02 22:48:31.310387 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:31.310355 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c8kg9" event={"ID":"72ef59a1-3734-47b7-9074-f683ca94dd32","Type":"ContainerStarted","Data":"686db8c7cbec860d262d53d72fe90fdaa466dc8bd1f9bfe8cc2e985c22812c6b"} Nov 02 22:48:31.310852 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:31.310446 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c8kg9" event={"ID":"72ef59a1-3734-47b7-9074-f683ca94dd32","Type":"ContainerStarted","Data":"2e113d5493704151ee3b3b6fadb3a1a065f173dec4c8aa4063ce558dc497ff69"} Nov 02 22:48:31.310852 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:31.310478 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-c8kg9" Nov 02 22:48:31.312590 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:31.312562 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-465bf" event={"ID":"82b3b5e2-09f1-49c0-8d72-c85187918223","Type":"ContainerStarted","Data":"6d87caff0029a6437ff2dc07bc49dc2e94d114dea8a9dceb9fc4c6cedecee0b2"} Nov 02 22:48:31.313929 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:31.313903 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qcz4m" event={"ID":"53ba063f-0db3-4f82-b280-40718c28b479","Type":"ContainerStarted","Data":"2b9e4b16553f7ea19145a13a43b5277ad0cd6e868501e01d6f6e0c3330dd2a9f"} Nov 02 22:48:31.315975 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:31.315956 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rf5x6" event={"ID":"2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc","Type":"ContainerStarted","Data":"7bd586b24a0e17bc9045508501e78b5c1b77ab0620cd1adc585f78a1a0962678"} Nov 02 22:48:31.316064 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:31.316009 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rf5x6" event={"ID":"2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc","Type":"ContainerStarted","Data":"31b76f7615e367a03693c47c9f59dfdc6522f7bd8fcae1e8c529ba6044afe22a"} Nov 02 22:48:31.436405 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:31.436355 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-c8kg9" podStartSLOduration=34.837170406 podStartE2EDuration="38.436341041s" totalImagesPullingTime="3.599170635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:47:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:31.434707911 +0000 UTC m=+69.997071249" watchObservedRunningTime="2026-11-02 22:48:31.436341041 +0000 UTC m=+69.998704378" Nov 02 22:48:31.550671 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:31.550611 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qcz4m" podStartSLOduration=413.586503968 podStartE2EDuration="6m57.550597999s" totalImagesPullingTime="3.964094031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:31.549066225 +0000 UTC m=+70.111429584" watchObservedRunningTime="2026-11-02 22:48:31.550597999 +0000 UTC m=+70.112961355" Nov 02 22:48:31.664608 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:31.664554 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-465bf" podStartSLOduration=67.413867677 podStartE2EDuration="1m9.664533592s" totalImagesPullingTime="2.250665915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:31.663369865 +0000 UTC m=+70.225733203" watchObservedRunningTime="2026-11-02 22:48:31.664533592 +0000 UTC m=+70.226896950" Nov 02 22:48:31.779485 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:31.779414 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rf5x6" podStartSLOduration=413.984421829 podStartE2EDuration="6m57.779394131s" totalImagesPullingTime="3.794972302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:31.778710564 +0000 UTC m=+70.341073902" watchObservedRunningTime="2026-11-02 22:48:31.779394131 +0000 UTC m=+70.341757469" Nov 02 22:48:33.184379 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:33.184350 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2qpkq" Nov 02 22:48:33.694160 ip-10-0-135-176 kubenswrapper[2573]: I1102 22:48:33.693533 2573 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 22:48:33.693730 ip-10-0-135-176 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 22:48:33.712564 ip-10-0-135-176 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 22:48:33.712816 ip-10-0-135-176 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 22:48:33.712880 ip-10-0-135-176 systemd[1]: kubelet.service: Consumed 4.523s CPU time. Nov 02 22:48:33.728752 ip-10-0-135-176 systemd[1]: Starting Kubernetes Kubelet... Nov 02 22:48:33.790785 ip-10-0-135-176 kubenswrapper[5727]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:48:33.790785 ip-10-0-135-176 kubenswrapper[5727]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 22:48:33.790785 ip-10-0-135-176 kubenswrapper[5727]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:48:33.790785 ip-10-0-135-176 kubenswrapper[5727]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:48:33.792821 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792739 5727 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:48:33.792821 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792821 5727 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:48:33.792821 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792825 5727 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:48:33.792923 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792828 5727 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:48:33.792923 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792831 5727 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:48:33.792923 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792835 5727 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:48:33.792923 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792839 5727 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:48:33.792923 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792842 5727 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:48:33.792923 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792844 5727 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:48:33.792923 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792847 5727 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:48:33.792923 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792849 5727 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:48:33.792923 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792852 5727 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:48:33.792923 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792854 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:48:33.792923 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792857 5727 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:48:33.792923 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792859 5727 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:48:33.792923 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792862 5727 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:48:33.792923 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792865 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:48:33.792923 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792867 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:48:33.792923 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792870 5727 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:48:33.792923 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792873 5727 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:48:33.792923 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792876 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:48:33.792923 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792878 5727 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:48:33.793430 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792881 5727 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:48:33.793430 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792883 5727 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:48:33.793430 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792886 5727 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:48:33.793430 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792889 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:48:33.793430 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792891 5727 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:48:33.793430 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792894 5727 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:48:33.793430 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792896 5727 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:48:33.793430 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792899 5727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:48:33.793430 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792901 5727 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:48:33.793430 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792904 5727 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:48:33.793430 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792907 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:48:33.793430 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792909 5727 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:48:33.793430 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792912 5727 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:48:33.793430 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792914 5727 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:48:33.793430 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792918 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:48:33.793430 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792921 5727 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:48:33.793430 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792923 5727 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:48:33.793430 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792931 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:48:33.793430 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792934 5727 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:48:33.793908 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792936 5727 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:48:33.793908 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792939 5727 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:48:33.793908 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792942 5727 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:48:33.793908 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792944 5727 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:48:33.793908 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792947 5727 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:48:33.793908 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792950 5727 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:48:33.793908 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792953 5727 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:48:33.793908 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792955 5727 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:48:33.793908 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792958 5727 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:48:33.793908 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792960 5727 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:48:33.793908 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792963 5727 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:48:33.793908 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792965 5727 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:48:33.793908 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792968 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:48:33.793908 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792970 5727 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:48:33.793908 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792972 5727 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:48:33.793908 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792975 5727 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:48:33.793908 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792978 5727 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:48:33.793908 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792980 5727 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:48:33.793908 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792983 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:48:33.793908 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792988 5727 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:48:33.794444 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792992 5727 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:48:33.794444 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.792997 5727 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:48:33.794444 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793000 5727 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:48:33.794444 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793002 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:48:33.794444 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793005 5727 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:48:33.794444 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793008 5727 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:48:33.794444 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793010 5727 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:48:33.794444 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793012 5727 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:48:33.794444 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793015 5727 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:48:33.794444 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793017 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:48:33.794444 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793020 5727 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:48:33.794444 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793023 5727 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:48:33.794444 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793025 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:48:33.794444 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793028 5727 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:48:33.794444 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793030 5727 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:48:33.794444 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793033 5727 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:48:33.794444 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793035 5727 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:48:33.794444 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793038 5727 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:48:33.794444 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793040 5727 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:48:33.794922 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793043 5727 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:48:33.794922 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793046 5727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:48:33.794922 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793048 5727 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:48:33.794922 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793051 5727 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:48:33.794922 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793054 5727 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:48:33.794922 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793056 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:48:33.794922 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793058 5727 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:48:33.794922 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793061 5727 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:48:33.794922 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793063 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:48:33.794922 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793066 5727 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:48:33.794922 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793069 5727 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:48:33.794922 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793071 5727 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:48:33.794922 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793073 5727 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:48:33.794922 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793076 5727 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:48:33.794922 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793078 5727 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:48:33.794922 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793081 5727 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:48:33.794922 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793083 5727 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:48:33.794922 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793086 5727 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:48:33.794922 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793088 5727 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:48:33.794922 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793090 5727 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:48:33.795543 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793093 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:48:33.795543 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793096 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:48:33.795543 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793099 5727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:48:33.795543 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793101 5727 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:48:33.795543 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793104 5727 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:48:33.795543 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793107 5727 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:48:33.795543 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793885 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:48:33.795543 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793892 5727 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:48:33.795543 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793895 5727 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:48:33.795543 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793897 5727 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:48:33.795543 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793901 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:48:33.795543 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793904 5727 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:48:33.795543 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793907 5727 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:48:33.795543 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793909 5727 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:48:33.795543 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793913 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:48:33.795543 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793915 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:48:33.795543 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793918 5727 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:48:33.795543 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793921 5727 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:48:33.795543 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793923 5727 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:48:33.796223 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793927 5727 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:48:33.796223 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793932 5727 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:48:33.796223 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793935 5727 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:48:33.796223 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793938 5727 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:48:33.796223 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793941 5727 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:48:33.796223 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793943 5727 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:48:33.796223 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793946 5727 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:48:33.796223 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793948 5727 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:48:33.796223 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793951 5727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:48:33.796223 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793953 5727 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:48:33.796223 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793956 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:48:33.796223 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793958 5727 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:48:33.796223 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793961 5727 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:48:33.796223 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793964 5727 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:48:33.796223 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793966 5727 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:48:33.796223 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793968 5727 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:48:33.796223 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793971 5727 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:48:33.796223 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793973 5727 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:48:33.796223 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793976 5727 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:48:33.796223 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793979 5727 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:48:33.796745 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793981 5727 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:48:33.796745 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793985 5727 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:48:33.796745 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793987 5727 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:48:33.796745 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793990 5727 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:48:33.796745 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793994 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:48:33.796745 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793997 5727 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:48:33.796745 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.793999 5727 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:48:33.796745 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794002 5727 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:48:33.796745 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794004 5727 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:48:33.796745 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794007 5727 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:48:33.796745 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794009 5727 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:48:33.796745 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794012 5727 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:48:33.796745 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794014 5727 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:48:33.796745 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794017 5727 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:48:33.796745 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794019 5727 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:48:33.796745 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794021 5727 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:48:33.796745 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794024 5727 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:48:33.796745 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794026 5727 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:48:33.796745 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794029 5727 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:48:33.796745 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794032 5727 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:48:33.797494 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794034 5727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:48:33.797494 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794037 5727 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:48:33.797494 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794040 5727 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:48:33.797494 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794043 5727 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:48:33.797494 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794045 5727 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:48:33.797494 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794048 5727 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:48:33.797494 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794050 5727 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:48:33.797494 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794053 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:48:33.797494 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794055 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:48:33.797494 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794058 5727 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:48:33.797494 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794060 5727 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:48:33.797494 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794063 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:48:33.797494 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794068 5727 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:48:33.797494 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794071 5727 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:48:33.797494 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794073 5727 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:48:33.797494 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794076 5727 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:48:33.797494 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794078 5727 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:48:33.797494 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794081 5727 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:48:33.797494 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794083 5727 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:48:33.797964 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794086 5727 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:48:33.797964 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794088 5727 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:48:33.797964 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794090 5727 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:48:33.797964 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794093 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:48:33.797964 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794095 5727 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:48:33.797964 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794098 5727 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:48:33.797964 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794100 5727 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:48:33.797964 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794103 5727 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:48:33.797964 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794106 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:48:33.797964 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794142 5727 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:48:33.797964 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794147 5727 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:48:33.797964 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794154 5727 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:48:33.797964 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794156 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:48:33.797964 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794159 5727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:48:33.797964 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794162 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:48:33.797964 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794165 5727 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:48:33.797964 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794168 5727 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:48:33.797964 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794171 5727 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:48:33.797964 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794173 5727 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:48:33.798463 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794176 5727 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:48:33.798463 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794178 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:48:33.798463 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794181 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:48:33.798463 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794183 5727 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:48:33.798463 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794186 5727 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:48:33.798463 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794188 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:48:33.798463 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794191 5727 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:48:33.798463 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794194 5727 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:48:33.798463 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794197 5727 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:48:33.798463 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794199 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:48:33.798463 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794201 5727 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:48:33.798463 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794204 5727 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:48:33.798463 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794208 5727 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:48:33.798463 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794212 5727 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:48:33.798463 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794215 5727 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:48:33.798463 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794706 5727 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:48:33.798463 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794711 5727 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:48:33.798463 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794715 5727 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:48:33.798463 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794718 5727 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:48:33.798987 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794721 5727 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:48:33.798987 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794724 5727 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:48:33.798987 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794726 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:48:33.798987 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794729 5727 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:48:33.798987 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794731 5727 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:48:33.798987 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794734 5727 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:48:33.798987 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794736 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:48:33.798987 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794740 5727 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:48:33.798987 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794743 5727 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:48:33.798987 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794747 5727 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:48:33.798987 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794749 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:48:33.798987 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794752 5727 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:48:33.798987 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794754 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:48:33.798987 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794757 5727 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:48:33.798987 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794760 5727 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:48:33.798987 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794762 5727 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:48:33.798987 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794765 5727 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:48:33.798987 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794767 5727 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:48:33.798987 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794770 5727 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:48:33.799506 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794772 5727 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:48:33.799506 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794780 5727 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:48:33.799506 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794783 5727 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:48:33.799506 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794785 5727 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:48:33.799506 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794788 5727 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:48:33.799506 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794790 5727 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:48:33.799506 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794794 5727 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:48:33.799506 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794798 5727 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:48:33.799506 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794801 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:48:33.799506 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794804 5727 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:48:33.799506 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794806 5727 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:48:33.799506 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794808 5727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:48:33.799506 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794811 5727 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:48:33.799506 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794813 5727 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:48:33.799506 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794816 5727 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:48:33.799506 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794819 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:48:33.799506 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794821 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:48:33.799506 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794823 5727 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:48:33.799506 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794826 5727 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:48:33.799969 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794829 5727 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:48:33.799969 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794831 5727 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:48:33.799969 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794834 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:48:33.799969 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794836 5727 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:48:33.799969 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794839 5727 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:48:33.799969 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794841 5727 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:48:33.799969 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794844 5727 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:48:33.799969 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794847 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:48:33.799969 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794849 5727 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:48:33.799969 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794852 5727 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:48:33.799969 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794854 5727 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:48:33.799969 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794856 5727 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:48:33.799969 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794859 5727 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:48:33.799969 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794861 5727 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:48:33.799969 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794864 5727 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:48:33.799969 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794867 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:48:33.799969 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794869 5727 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:48:33.799969 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794872 5727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:48:33.799969 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794874 5727 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:48:33.800462 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794877 5727 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:48:33.800462 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794879 5727 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:48:33.800462 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794881 5727 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:48:33.800462 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794883 5727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:48:33.800462 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794889 5727 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:48:33.800462 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794891 5727 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:48:33.800462 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794894 5727 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:48:33.800462 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794896 5727 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:48:33.800462 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794899 5727 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:48:33.800462 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794901 5727 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:48:33.800462 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794904 5727 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:48:33.800462 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794908 5727 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:48:33.800462 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794910 5727 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:48:33.800462 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794913 5727 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:48:33.800462 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794916 5727 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:48:33.800462 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794918 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:48:33.800462 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794921 5727 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:48:33.800462 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794924 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:48:33.800462 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794926 5727 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:48:33.800462 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794929 5727 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:48:33.800957 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794932 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:48:33.800957 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794934 5727 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:48:33.800957 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794937 5727 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:48:33.800957 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794940 5727 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:48:33.800957 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794942 5727 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:48:33.800957 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794945 5727 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:48:33.800957 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794947 5727 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:48:33.800957 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794950 5727 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:48:33.800957 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794952 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:48:33.800957 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794955 5727 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:48:33.800957 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794957 5727 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:48:33.800957 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794960 5727 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:48:33.800957 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794962 5727 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:48:33.800957 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794965 5727 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:48:33.800957 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794968 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:48:33.800957 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794970 5727 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:48:33.800957 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794974 5727 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:48:33.800957 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794977 5727 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:48:33.800957 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794980 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:48:33.800957 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794982 5727 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:48:33.801759 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794984 5727 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:48:33.801759 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794987 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:48:33.801759 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794989 5727 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:48:33.801759 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794993 5727 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:48:33.801759 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.794996 5727 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:48:33.801759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795100 5727 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 22:48:33.801759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795111 5727 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 22:48:33.801759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795120 5727 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 22:48:33.801759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795141 5727 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 22:48:33.801759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795146 5727 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 22:48:33.801759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795149 5727 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 22:48:33.801759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795153 5727 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 22:48:33.801759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795157 5727 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 22:48:33.801759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795160 5727 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 22:48:33.801759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795163 5727 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 22:48:33.801759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795166 5727 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 22:48:33.801759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795169 5727 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 22:48:33.801759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795172 5727 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 22:48:33.801759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795175 5727 flags.go:64] FLAG: --cgroup-root="" Nov 02 22:48:33.801759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795178 5727 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 22:48:33.801759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795181 5727 flags.go:64] FLAG: --client-ca-file="" Nov 02 22:48:33.801759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795183 5727 flags.go:64] FLAG: --cloud-provider="external" Nov 02 22:48:33.801759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795193 5727 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 22:48:33.802548 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795197 5727 flags.go:64] FLAG: --cluster-domain="" Nov 02 22:48:33.802548 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795200 5727 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 22:48:33.802548 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795203 5727 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 22:48:33.802548 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795206 5727 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 22:48:33.802548 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795210 5727 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 22:48:33.802548 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795216 5727 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 22:48:33.802548 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795218 5727 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 22:48:33.802548 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795221 5727 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 22:48:33.802548 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795225 5727 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 22:48:33.802548 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795227 5727 flags.go:64] FLAG: --contention-profiling="false" Nov 02 22:48:33.802548 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795230 5727 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 22:48:33.802548 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795233 5727 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 22:48:33.802548 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795238 5727 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 22:48:33.802548 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795241 5727 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 22:48:33.802548 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795245 5727 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 22:48:33.802548 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795248 5727 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 22:48:33.802548 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795251 5727 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 22:48:33.802548 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795254 5727 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 22:48:33.802548 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795257 5727 flags.go:64] FLAG: --enable-server="true" Nov 02 22:48:33.802548 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795260 5727 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 22:48:33.802548 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795267 5727 flags.go:64] FLAG: --event-burst="100" Nov 02 22:48:33.802548 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795270 5727 flags.go:64] FLAG: --event-qps="50" Nov 02 22:48:33.802548 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795273 5727 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 22:48:33.802548 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795276 5727 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 22:48:33.803171 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795279 5727 flags.go:64] FLAG: --eviction-hard="" Nov 02 22:48:33.803171 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795283 5727 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 22:48:33.803171 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795286 5727 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 22:48:33.803171 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795289 5727 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 22:48:33.803171 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795292 5727 flags.go:64] FLAG: --eviction-soft="" Nov 02 22:48:33.803171 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795295 5727 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 22:48:33.803171 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795298 5727 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 22:48:33.803171 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795300 5727 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 22:48:33.803171 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795309 5727 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 22:48:33.803171 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795312 5727 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 22:48:33.803171 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795315 5727 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 22:48:33.803171 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795318 5727 flags.go:64] FLAG: --feature-gates="" Nov 02 22:48:33.803171 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795323 5727 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 22:48:33.803171 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795327 5727 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 22:48:33.803171 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795330 5727 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 22:48:33.803171 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795333 5727 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 22:48:33.803171 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795336 5727 flags.go:64] FLAG: --healthz-port="10248" Nov 02 22:48:33.803171 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795339 5727 flags.go:64] FLAG: --help="false" Nov 02 22:48:33.803171 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795341 5727 flags.go:64] FLAG: --hostname-override="ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:48:33.803171 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795344 5727 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 22:48:33.803171 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795349 5727 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 22:48:33.803171 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795352 5727 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 22:48:33.803171 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795355 5727 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 22:48:33.803759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795359 5727 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 22:48:33.803759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795362 5727 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 22:48:33.803759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795364 5727 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 22:48:33.803759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795367 5727 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 22:48:33.803759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795370 5727 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 22:48:33.803759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795373 5727 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 22:48:33.803759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795376 5727 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 22:48:33.803759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795379 5727 flags.go:64] FLAG: --kube-reserved="" Nov 02 22:48:33.803759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795381 5727 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 22:48:33.803759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795384 5727 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 22:48:33.803759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795387 5727 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 22:48:33.803759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795390 5727 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 22:48:33.803759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795393 5727 flags.go:64] FLAG: --lock-file="" Nov 02 22:48:33.803759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795395 5727 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 22:48:33.803759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795398 5727 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 22:48:33.803759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795401 5727 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 22:48:33.803759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795407 5727 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 22:48:33.803759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795409 5727 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 22:48:33.803759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795418 5727 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 22:48:33.803759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795427 5727 flags.go:64] FLAG: --logging-format="text" Nov 02 22:48:33.803759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795431 5727 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 22:48:33.803759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795434 5727 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 22:48:33.803759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795438 5727 flags.go:64] FLAG: --manifest-url="" Nov 02 22:48:33.803759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795441 5727 flags.go:64] FLAG: --manifest-url-header="" Nov 02 22:48:33.803759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795446 5727 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 22:48:33.804504 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795449 5727 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 22:48:33.804504 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795453 5727 flags.go:64] FLAG: --max-pods="110" Nov 02 22:48:33.804504 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795456 5727 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 22:48:33.804504 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795459 5727 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 22:48:33.804504 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795463 5727 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 22:48:33.804504 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795466 5727 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 22:48:33.804504 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795469 5727 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 22:48:33.804504 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795472 5727 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 22:48:33.804504 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795475 5727 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 22:48:33.804504 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795485 5727 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 22:48:33.804504 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795489 5727 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 22:48:33.804504 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795494 5727 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 22:48:33.804504 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795498 5727 flags.go:64] FLAG: --pod-cidr="" Nov 02 22:48:33.804504 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795502 5727 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 22:48:33.804504 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795505 5727 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 22:48:33.804504 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795510 5727 flags.go:64] FLAG: --pods-per-core="0" Nov 02 22:48:33.804504 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795514 5727 flags.go:64] FLAG: --port="10250" Nov 02 22:48:33.804504 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795518 5727 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 22:48:33.804504 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795522 5727 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0fed1cfa20bbd3c95" Nov 02 22:48:33.804504 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795526 5727 flags.go:64] FLAG: --qos-reserved="" Nov 02 22:48:33.804504 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795531 5727 flags.go:64] FLAG: --read-only-port="10255" Nov 02 22:48:33.804504 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795535 5727 flags.go:64] FLAG: --register-node="true" Nov 02 22:48:33.804504 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795539 5727 flags.go:64] FLAG: --register-with-taints="" Nov 02 22:48:33.804504 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795545 5727 flags.go:64] FLAG: --registry-burst="10" Nov 02 22:48:33.804504 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795550 5727 flags.go:64] FLAG: --registry-qps="5" Nov 02 22:48:33.804504 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795554 5727 flags.go:64] FLAG: --reserved-cpus="" Nov 02 22:48:33.805167 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795570 5727 flags.go:64] FLAG: --reserved-memory="" Nov 02 22:48:33.805167 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795576 5727 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 22:48:33.805167 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795581 5727 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 22:48:33.805167 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795591 5727 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 22:48:33.805167 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795596 5727 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 22:48:33.805167 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795600 5727 flags.go:64] FLAG: --runonce="false" Nov 02 22:48:33.805167 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795605 5727 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 22:48:33.805167 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795610 5727 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 22:48:33.805167 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795614 5727 flags.go:64] FLAG: --seccomp-default="false" Nov 02 22:48:33.805167 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795619 5727 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 22:48:33.805167 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795625 5727 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 22:48:33.805167 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795630 5727 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 22:48:33.805167 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795633 5727 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 22:48:33.805167 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795635 5727 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 22:48:33.805167 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795638 5727 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 22:48:33.805167 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795641 5727 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 22:48:33.805167 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795644 5727 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 22:48:33.805167 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795646 5727 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 22:48:33.805167 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795649 5727 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 22:48:33.805167 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795652 5727 flags.go:64] FLAG: --system-cgroups="" Nov 02 22:48:33.805167 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795655 5727 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 22:48:33.805167 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795661 5727 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 22:48:33.805167 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795663 5727 flags.go:64] FLAG: --tls-cert-file="" Nov 02 22:48:33.805167 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795666 5727 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 22:48:33.805167 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795677 5727 flags.go:64] FLAG: --tls-min-version="" Nov 02 22:48:33.805789 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795686 5727 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 22:48:33.805789 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795689 5727 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 22:48:33.805789 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795692 5727 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 22:48:33.805789 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795694 5727 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 22:48:33.805789 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795697 5727 flags.go:64] FLAG: --v="2" Nov 02 22:48:33.805789 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795702 5727 flags.go:64] FLAG: --version="false" Nov 02 22:48:33.805789 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795708 5727 flags.go:64] FLAG: --vmodule="" Nov 02 22:48:33.805789 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795715 5727 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 22:48:33.805789 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.795728 5727 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 22:48:33.805789 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.795944 5727 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:48:33.805789 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.795952 5727 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:48:33.805789 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.795956 5727 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:48:33.805789 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.795959 5727 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:48:33.805789 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.795962 5727 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:48:33.805789 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.795964 5727 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:48:33.805789 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.795968 5727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:48:33.805789 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.795970 5727 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:48:33.805789 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.795973 5727 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:48:33.805789 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.795975 5727 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:48:33.805789 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.795978 5727 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:48:33.805789 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.795980 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:48:33.805789 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.795983 5727 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:48:33.806357 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.795986 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:48:33.806357 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.795988 5727 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:48:33.806357 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.795991 5727 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:48:33.806357 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.795993 5727 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:48:33.806357 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.795996 5727 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:48:33.806357 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.795999 5727 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:48:33.806357 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796001 5727 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:48:33.806357 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796004 5727 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:48:33.806357 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796006 5727 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:48:33.806357 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796009 5727 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:48:33.806357 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796011 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:48:33.806357 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796014 5727 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:48:33.806357 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796018 5727 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:48:33.806357 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796022 5727 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:48:33.806357 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796024 5727 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:48:33.806357 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796028 5727 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:48:33.806357 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796032 5727 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:48:33.806357 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796034 5727 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:48:33.806357 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796037 5727 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:48:33.806865 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796045 5727 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:48:33.806865 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796050 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:48:33.806865 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796053 5727 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:48:33.806865 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796056 5727 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:48:33.806865 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796059 5727 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:48:33.806865 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796062 5727 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:48:33.806865 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796065 5727 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:48:33.806865 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796068 5727 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:48:33.806865 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796071 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:48:33.806865 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796073 5727 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:48:33.806865 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796076 5727 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:48:33.806865 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796078 5727 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:48:33.806865 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796081 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:48:33.806865 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796084 5727 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:48:33.806865 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796086 5727 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:48:33.806865 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796089 5727 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:48:33.806865 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796091 5727 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:48:33.806865 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796094 5727 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:48:33.806865 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796097 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:48:33.806865 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796099 5727 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:48:33.807474 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796101 5727 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:48:33.807474 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796104 5727 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:48:33.807474 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796106 5727 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:48:33.807474 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796109 5727 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:48:33.807474 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796111 5727 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:48:33.807474 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796114 5727 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:48:33.807474 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796116 5727 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:48:33.807474 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796119 5727 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:48:33.807474 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796121 5727 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:48:33.807474 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796139 5727 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:48:33.807474 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796142 5727 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:48:33.807474 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796145 5727 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:48:33.807474 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796147 5727 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:48:33.807474 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796157 5727 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:48:33.807474 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796160 5727 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:48:33.807474 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796163 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:48:33.807474 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796165 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:48:33.807474 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796168 5727 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:48:33.807474 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796170 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:48:33.808143 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796172 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:48:33.808143 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796175 5727 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:48:33.808143 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796178 5727 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:48:33.808143 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796180 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:48:33.808143 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796183 5727 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:48:33.808143 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796185 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:48:33.808143 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796188 5727 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:48:33.808143 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796190 5727 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:48:33.808143 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796193 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:48:33.808143 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796195 5727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:48:33.808143 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796198 5727 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:48:33.808143 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796200 5727 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:48:33.808143 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796205 5727 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:48:33.808143 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796207 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:48:33.808143 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796210 5727 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:48:33.808143 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796212 5727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:48:33.808143 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796215 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:48:33.808143 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796218 5727 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:48:33.808143 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796220 5727 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:48:33.808143 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796223 5727 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:48:33.809057 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796226 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:48:33.809057 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796228 5727 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:48:33.809057 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796231 5727 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:48:33.809057 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796234 5727 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:48:33.809057 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796236 5727 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:48:33.809057 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796239 5727 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:48:33.809057 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796244 5727 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:48:33.809057 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796252 5727 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:48:33.809057 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796255 5727 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:48:33.809057 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796257 5727 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:48:33.809057 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796260 5727 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:48:33.809057 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796263 5727 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:48:33.809057 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796265 5727 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:48:33.809057 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796268 5727 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:48:33.809057 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796270 5727 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:48:33.809707 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.796299 5727 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:48:33.809707 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796347 5727 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:48:33.809707 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796352 5727 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:48:33.809707 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796355 5727 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:48:33.809707 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796357 5727 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:48:33.809707 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796360 5727 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:48:33.809707 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796363 5727 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:48:33.809707 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796366 5727 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:48:33.809707 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796369 5727 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:48:33.809707 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796373 5727 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:48:33.809707 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796376 5727 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:48:33.809707 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796378 5727 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:48:33.809707 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796380 5727 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:48:33.809707 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796383 5727 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:48:33.809707 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796385 5727 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:48:33.810236 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796388 5727 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:48:33.810236 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796390 5727 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:48:33.810236 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796393 5727 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:48:33.810236 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796395 5727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:48:33.810236 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796398 5727 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:48:33.810236 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796400 5727 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:48:33.810236 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796403 5727 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:48:33.810236 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796405 5727 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:48:33.810236 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796408 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:48:33.810236 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796411 5727 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:48:33.810236 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796419 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:48:33.810236 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796422 5727 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:48:33.810236 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796425 5727 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:48:33.810236 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796427 5727 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:48:33.810236 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796430 5727 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:48:33.810236 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796433 5727 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:48:33.810236 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796435 5727 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:48:33.810236 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796438 5727 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:48:33.810236 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796440 5727 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:48:33.810236 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796446 5727 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:48:33.810852 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796449 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:48:33.810852 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796451 5727 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:48:33.810852 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796455 5727 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:48:33.810852 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796458 5727 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:48:33.810852 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796461 5727 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:48:33.810852 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796465 5727 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:48:33.810852 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796469 5727 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:48:33.810852 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796472 5727 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:48:33.810852 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796475 5727 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:48:33.810852 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796478 5727 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:48:33.810852 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796480 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:48:33.810852 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796483 5727 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:48:33.810852 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796485 5727 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:48:33.810852 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796488 5727 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:48:33.810852 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796490 5727 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:48:33.810852 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796492 5727 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:48:33.810852 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796495 5727 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:48:33.810852 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796497 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:48:33.810852 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796500 5727 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:48:33.811415 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796502 5727 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:48:33.811415 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796504 5727 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:48:33.811415 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796508 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:48:33.811415 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796511 5727 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:48:33.811415 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796515 5727 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:48:33.811415 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796518 5727 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:48:33.811415 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796520 5727 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:48:33.811415 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796523 5727 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:48:33.811415 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796525 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:48:33.811415 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796528 5727 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:48:33.811415 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796530 5727 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:48:33.811415 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796532 5727 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:48:33.811415 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796535 5727 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:48:33.811415 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796537 5727 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:48:33.811415 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796540 5727 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:48:33.811415 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796542 5727 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:48:33.811415 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796545 5727 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:48:33.811415 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796547 5727 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:48:33.811415 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796549 5727 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:48:33.811415 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796553 5727 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:48:33.811925 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796556 5727 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:48:33.811925 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796558 5727 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:48:33.811925 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796561 5727 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:48:33.811925 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796563 5727 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:48:33.811925 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796565 5727 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:48:33.811925 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796568 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:48:33.811925 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796570 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:48:33.811925 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796573 5727 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:48:33.811925 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796575 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:48:33.811925 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796578 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:48:33.811925 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796580 5727 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:48:33.811925 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796582 5727 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:48:33.811925 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796585 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:48:33.811925 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796587 5727 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:48:33.811925 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796591 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:48:33.811925 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796593 5727 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:48:33.811925 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796596 5727 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:48:33.811925 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796605 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:48:33.811925 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796607 5727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:48:33.812409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796610 5727 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:48:33.812409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796613 5727 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:48:33.812409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796615 5727 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:48:33.812409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796618 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:48:33.812409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796620 5727 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:48:33.812409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796623 5727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:48:33.812409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796625 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:48:33.812409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796628 5727 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:48:33.812409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796630 5727 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:48:33.812409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796633 5727 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:48:33.812409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796635 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:48:33.812409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796637 5727 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:48:33.812409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796641 5727 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:48:33.812409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.796644 5727 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:48:33.812409 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.800663 5727 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 22:48:33.812409 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.800681 5727 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 22:48:33.812409 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.800703 5727 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 22:48:33.812409 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.800708 5727 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 22:48:33.812409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800758 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:48:33.812869 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800763 5727 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:48:33.812869 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800766 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:48:33.812869 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800769 5727 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:48:33.812869 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800772 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:48:33.812869 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800775 5727 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:48:33.812869 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800777 5727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:48:33.812869 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800780 5727 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:48:33.812869 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800783 5727 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:48:33.812869 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800785 5727 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:48:33.812869 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800788 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:48:33.812869 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800790 5727 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:48:33.812869 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800793 5727 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:48:33.812869 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800796 5727 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:48:33.812869 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800798 5727 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:48:33.812869 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800801 5727 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:48:33.812869 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800803 5727 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:48:33.812869 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800806 5727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:48:33.812869 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800808 5727 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:48:33.812869 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800811 5727 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:48:33.813432 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800813 5727 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:48:33.813432 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800815 5727 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:48:33.813432 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800818 5727 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:48:33.813432 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800821 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:48:33.813432 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800824 5727 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:48:33.813432 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800827 5727 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:48:33.813432 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800829 5727 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:48:33.813432 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800832 5727 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:48:33.813432 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800834 5727 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:48:33.813432 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800838 5727 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:48:33.813432 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800842 5727 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:48:33.813432 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800845 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:48:33.813432 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800847 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:48:33.813432 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800850 5727 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:48:33.813432 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800852 5727 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:48:33.813432 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800855 5727 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:48:33.813432 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800857 5727 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:48:33.813432 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800860 5727 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:48:33.813432 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800863 5727 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:48:33.814060 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800865 5727 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:48:33.814060 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800868 5727 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:48:33.814060 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800870 5727 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:48:33.814060 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800873 5727 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:48:33.814060 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800875 5727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:48:33.814060 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800878 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:48:33.814060 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800881 5727 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:48:33.814060 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800883 5727 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:48:33.814060 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800886 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:48:33.814060 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800888 5727 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:48:33.814060 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800891 5727 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:48:33.814060 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800893 5727 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:48:33.814060 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800895 5727 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:48:33.814060 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800898 5727 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:48:33.814060 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800900 5727 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:48:33.814060 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800903 5727 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:48:33.814060 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800905 5727 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:48:33.814060 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800908 5727 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:48:33.814060 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800910 5727 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:48:33.814060 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800912 5727 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:48:33.814771 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800915 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:48:33.814771 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800918 5727 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:48:33.814771 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800920 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:48:33.814771 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800923 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:48:33.814771 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800926 5727 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:48:33.814771 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800928 5727 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:48:33.814771 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800931 5727 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:48:33.814771 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800933 5727 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:48:33.814771 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800935 5727 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:48:33.814771 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800938 5727 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:48:33.814771 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800940 5727 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:48:33.814771 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800943 5727 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:48:33.814771 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800945 5727 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:48:33.814771 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800948 5727 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:48:33.814771 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800950 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:48:33.814771 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800953 5727 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:48:33.814771 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800955 5727 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:48:33.814771 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800957 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:48:33.814771 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800960 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:48:33.814771 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800963 5727 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:48:33.815460 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800966 5727 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:48:33.815460 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800968 5727 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:48:33.815460 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800971 5727 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:48:33.815460 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800974 5727 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:48:33.815460 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800976 5727 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:48:33.815460 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800978 5727 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:48:33.815460 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800981 5727 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:48:33.815460 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800983 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:48:33.815460 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800986 5727 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:48:33.815460 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800988 5727 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:48:33.815460 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800991 5727 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:48:33.815460 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800993 5727 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:48:33.815460 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800996 5727 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:48:33.815460 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.800998 5727 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:48:33.815460 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.801001 5727 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:48:33.815460 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.801003 5727 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:48:33.815460 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.801006 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:48:33.815460 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.801009 5727 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:48:33.815460 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.801011 5727 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:48:33.815460 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.801014 5727 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:48:33.815975 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.801016 5727 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:48:33.815975 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.801020 5727 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:48:33.815975 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.801024 5727 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:48:33.815975 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.801028 5727 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:48:33.815975 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.801030 5727 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:48:33.815975 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.801033 5727 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:48:33.815975 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.801036 5727 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:48:33.815975 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.801063 5727 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:48:33.815975 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802217 5727 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:48:33.815975 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802239 5727 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:48:33.815975 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802247 5727 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:48:33.815975 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802253 5727 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:48:33.815975 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802263 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:48:33.815975 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802289 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:48:33.815975 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802296 5727 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:48:33.816409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802388 5727 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:48:33.816409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802396 5727 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:48:33.816409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802403 5727 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:48:33.816409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802409 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:48:33.816409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802413 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:48:33.816409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802418 5727 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:48:33.816409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802422 5727 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:48:33.816409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802426 5727 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:48:33.816409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802430 5727 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:48:33.816409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802434 5727 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:48:33.816409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802438 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:48:33.816409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802442 5727 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:48:33.816409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802446 5727 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:48:33.816409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802451 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:48:33.816409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802455 5727 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:48:33.816409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802459 5727 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:48:33.816409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802463 5727 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:48:33.816409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802467 5727 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:48:33.816409 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802471 5727 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:48:33.816879 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802475 5727 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:48:33.816879 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802479 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:48:33.816879 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802484 5727 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:48:33.816879 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802488 5727 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:48:33.816879 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802494 5727 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:48:33.816879 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802499 5727 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:48:33.816879 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802503 5727 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:48:33.816879 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802508 5727 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:48:33.816879 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802512 5727 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:48:33.816879 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802516 5727 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:48:33.816879 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802520 5727 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:48:33.816879 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802524 5727 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:48:33.816879 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802528 5727 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:48:33.816879 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802532 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:48:33.816879 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802536 5727 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:48:33.816879 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802540 5727 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:48:33.816879 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802544 5727 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:48:33.816879 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802548 5727 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:48:33.816879 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802552 5727 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:48:33.816879 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802556 5727 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:48:33.817386 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802561 5727 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:48:33.817386 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802565 5727 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:48:33.817386 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802569 5727 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:48:33.817386 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802574 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:48:33.817386 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802579 5727 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:48:33.817386 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802583 5727 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:48:33.817386 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802590 5727 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:48:33.817386 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802597 5727 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:48:33.817386 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802602 5727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:48:33.817386 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802606 5727 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:48:33.817386 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802611 5727 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:48:33.817386 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802615 5727 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:48:33.817386 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802617 5727 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:48:33.817386 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802620 5727 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:48:33.817386 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802622 5727 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:48:33.817386 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802625 5727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:48:33.817386 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802628 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:48:33.817386 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802631 5727 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:48:33.817386 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802634 5727 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:48:33.817887 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802636 5727 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:48:33.817887 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802639 5727 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:48:33.817887 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802642 5727 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:48:33.817887 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802644 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:48:33.817887 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802647 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:48:33.817887 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802650 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:48:33.817887 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802652 5727 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:48:33.817887 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802655 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:48:33.817887 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802657 5727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:48:33.817887 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802660 5727 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:48:33.817887 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802662 5727 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:48:33.817887 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802664 5727 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:48:33.817887 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802667 5727 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:48:33.817887 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802669 5727 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:48:33.817887 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802672 5727 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:48:33.817887 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802674 5727 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:48:33.817887 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802677 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:48:33.817887 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802680 5727 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:48:33.817887 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802690 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:48:33.818416 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802693 5727 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:48:33.818416 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802695 5727 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:48:33.818416 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802698 5727 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:48:33.818416 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802700 5727 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:48:33.818416 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802703 5727 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:48:33.818416 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802705 5727 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:48:33.818416 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802708 5727 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:48:33.818416 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802710 5727 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:48:33.818416 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802713 5727 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:48:33.818416 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802716 5727 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:48:33.818416 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802718 5727 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:48:33.818416 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802720 5727 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:48:33.818416 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802723 5727 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:48:33.818416 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802725 5727 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:48:33.818416 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802728 5727 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:48:33.818416 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802730 5727 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:48:33.818416 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802732 5727 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:48:33.818416 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802735 5727 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:48:33.818416 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802737 5727 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:48:33.818416 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802740 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:48:33.818893 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802742 5727 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:48:33.818893 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802744 5727 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:48:33.818893 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.802777 5727 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:48:33.818893 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802813 5727 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:48:33.818893 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802818 5727 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:48:33.818893 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802822 5727 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:48:33.818893 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802825 5727 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:48:33.818893 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802828 5727 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:48:33.818893 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802830 5727 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:48:33.818893 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802833 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:48:33.818893 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802835 5727 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:48:33.818893 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802838 5727 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:48:33.818893 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802847 5727 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:48:33.818893 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802850 5727 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:48:33.818893 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802853 5727 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:48:33.818893 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802856 5727 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:48:33.819311 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802858 5727 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:48:33.819311 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802860 5727 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:48:33.819311 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802863 5727 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:48:33.819311 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802865 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:48:33.819311 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802868 5727 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:48:33.819311 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802870 5727 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:48:33.819311 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802874 5727 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:48:33.819311 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802878 5727 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:48:33.819311 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802881 5727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:48:33.819311 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802884 5727 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:48:33.819311 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802888 5727 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:48:33.819311 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802891 5727 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:48:33.819311 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802893 5727 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:48:33.819311 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802896 5727 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:48:33.819311 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802898 5727 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:48:33.819311 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802901 5727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:48:33.819311 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802903 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:48:33.819311 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802906 5727 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:48:33.819311 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802908 5727 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:48:33.819778 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802911 5727 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:48:33.819778 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802914 5727 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:48:33.819778 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802917 5727 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:48:33.819778 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802920 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:48:33.819778 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802922 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:48:33.819778 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802925 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:48:33.819778 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802927 5727 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:48:33.819778 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802930 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:48:33.819778 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802933 5727 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:48:33.819778 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802935 5727 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:48:33.819778 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802943 5727 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:48:33.819778 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802946 5727 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:48:33.819778 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802949 5727 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:48:33.819778 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802951 5727 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:48:33.819778 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802953 5727 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:48:33.819778 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802956 5727 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:48:33.819778 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802959 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:48:33.819778 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802961 5727 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:48:33.819778 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802964 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:48:33.820263 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802966 5727 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:48:33.820263 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802968 5727 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:48:33.820263 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802971 5727 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:48:33.820263 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802974 5727 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:48:33.820263 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802976 5727 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:48:33.820263 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802978 5727 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:48:33.820263 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802981 5727 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:48:33.820263 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802983 5727 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:48:33.820263 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802986 5727 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:48:33.820263 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802989 5727 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:48:33.820263 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802991 5727 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:48:33.820263 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802994 5727 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:48:33.820263 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802996 5727 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:48:33.820263 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.802999 5727 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:48:33.820263 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803002 5727 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:48:33.820263 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803004 5727 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:48:33.820263 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803007 5727 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:48:33.820263 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803009 5727 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:48:33.820263 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803012 5727 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:48:33.820263 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803014 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:48:33.820805 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803017 5727 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:48:33.820805 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803019 5727 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:48:33.820805 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803022 5727 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:48:33.820805 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803025 5727 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:48:33.820805 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803033 5727 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:48:33.820805 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803036 5727 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:48:33.820805 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803039 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:48:33.820805 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803041 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:48:33.820805 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803048 5727 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:48:33.820805 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803051 5727 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:48:33.820805 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803055 5727 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:48:33.820805 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803058 5727 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:48:33.820805 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803062 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:48:33.820805 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803065 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:48:33.820805 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803068 5727 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:48:33.820805 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803070 5727 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:48:33.820805 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803073 5727 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:48:33.820805 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803076 5727 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:48:33.820805 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803078 5727 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:48:33.821367 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803081 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:48:33.821367 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803083 5727 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:48:33.821367 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803086 5727 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:48:33.821367 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803088 5727 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:48:33.821367 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803091 5727 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:48:33.821367 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803093 5727 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:48:33.821367 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803096 5727 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:48:33.821367 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803098 5727 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:48:33.821367 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803101 5727 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:48:33.821367 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803103 5727 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:48:33.821367 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803106 5727 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:48:33.821367 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803110 5727 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:48:33.821367 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803112 5727 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:48:33.821367 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803115 5727 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:48:33.821367 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803117 5727 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:48:33.821367 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:33.803120 5727 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:48:33.821367 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.803195 5727 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 22:48:33.821367 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.804208 5727 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 22:48:33.821367 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.804261 5727 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 22:48:33.822146 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.804460 5727 server.go:1014] "Starting client certificate rotation" Nov 02 22:48:33.822146 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.804599 5727 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 22:48:33.822146 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.804643 5727 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 22:42:22 +0000 UTC" deadline="2028-05-30 01:10:23.455900963 +0000 UTC" Nov 02 22:48:33.822146 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.804672 5727 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13778h21m49.651230724s" Nov 02 22:48:33.822146 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.804959 5727 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 22:48:33.822146 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.806103 5727 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 22:48:33.822146 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.807335 5727 log.go:25] "Validated CRI v1 runtime API" Nov 02 22:48:33.822146 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.809737 5727 log.go:25] "Validated CRI v1 image API" Nov 02 22:48:33.822146 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.810521 5727 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 22:48:33.822146 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.812797 5727 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 aa01b5c2-a734-4ab5-97a2-b17b8799dbc8:/dev/nvme0n1p3 e9619993-14af-4bf9-85f5-a712396b498f:/dev/nvme0n1p4] Nov 02 22:48:33.822651 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.812816 5727 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/061fd09ec7499698a85f2e9bb065f4baa4da0ef59ed108bd060b11ee0c4b7d9e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/061fd09ec7499698a85f2e9bb065f4baa4da0ef59ed108bd060b11ee0c4b7d9e/userdata/shm major:0 minor:412 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/12d0b5f3c4797a1b4b8958ae37822899ad7f836fe7527324eff473eb7eee945b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/12d0b5f3c4797a1b4b8958ae37822899ad7f836fe7527324eff473eb7eee945b/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/149bfac297d1959995d6a77e8d0cb2e89a62a5efe665a352f8dd48c6a210de40/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/149bfac297d1959995d6a77e8d0cb2e89a62a5efe665a352f8dd48c6a210de40/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1dbff91d6a416a7a712a6415f02a9e2f4fb91cbbeeacad097a2516b6781d2736/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1dbff91d6a416a7a712a6415f02a9e2f4fb91cbbeeacad097a2516b6781d2736/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2035e8f997a9e7d0935f37f90194f850ae3c24b21829a8b22471026bfae92ce8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2035e8f997a9e7d0935f37f90194f850ae3c24b21829a8b22471026bfae92ce8/userdata/shm major:0 minor:379 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/20aeb9b1286b40414ad4b557e23c6578bd0da566586adfb27316b45cd1632aef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/20aeb9b1286b40414ad4b557e23c6578bd0da566586adfb27316b45cd1632aef/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2393b59785260194d19a54699f33e597481d2bf7ebd4c3b1679768477af03b4e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2393b59785260194d19a54699f33e597481d2bf7ebd4c3b1679768477af03b4e/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2762b05b8165a3da012d5281faadff367ecfd35437f6345d467e82f78916402a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2762b05b8165a3da012d5281faadff367ecfd35437f6345d467e82f78916402a/userdata/shm major:0 minor:377 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/278cf9de5cdac5e81c0f35ddc4d882c8f4fd7f0a7a2cb7f4717d54c016007ec0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/278cf9de5cdac5e81c0f35ddc4d882c8f4fd7f0a7a2cb7f4717d54c016007ec0/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d0f15b28f79d9d966377869a930b9cd17054031a166bfbbd0473f25097a1d2a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d0f15b28f79d9d966377869a930b9cd17054031a166bfbbd0473f25097a1d2a/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2f5f80417d312f9989700350646e7b60429b5eb784cc4e0e275da19371dbc8d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2f5f80417d312f9989700350646e7b60429b5eb784cc4e0e275da19371dbc8d3/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31b84ffae7960055d9afb0250f2959a11bbbc6bd0d76c87a2c5a2c9021b420e0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31b84ffae7960055d9afb0250f2959a11bbbc6bd0d76c87a2c5a2c9021b420e0/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/320f557ca944b584051436e7f2df20c9bdd300bcbb801c8a193fc912ffec0505/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/320f557ca944b584051436e7f2df20c9bdd300bcbb801c8a193fc912ffec0505/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4a29976e47440460e64c2e170cbc823f58c5efb479a898fb03278815767a5c2b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4a29976e47440460e64c2e170cbc823f58c5efb479a898fb03278815767a5c2b/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e20810858ab32fdce1dd75df90c3dbfc2084c6aca8fa0711e884db900522380/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e20810858ab32fdce1dd75df90c3dbfc2084c6aca8fa0711e884db900522380/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4fba275614bd2ddaf44e9966d406d568f3b834031d711e5499040da2ca9c527d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4fba275614bd2ddaf44e9966d406d568f3b834031d711e5499040da2ca9c527d/userdata/shm major:0 minor:381 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5399dea54dbec22dce782d3b4c70b29d1f0dfd0191efc4025948b94dc5a9cbd9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5399dea54dbec22dce782d3b4c70b29d1f0dfd0191efc4025948b94dc5a9cbd9/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/57a51cd7c0976e7981c9f7af8100910430f2502147ec3d8f927b084372b98d6f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/57a51cd7c0976e7981c9f7af8100910430f2502147ec3d8f927b084372b98d6f/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5ed5ea0522eca41e9753ad49cca68438d6ba6ae49d49745bcc1487e938928f84/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5ed5ea0522eca41e9753ad49cca68438d6ba6ae49d49745bcc1487e938928f84/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6002186f05dc05c6ae18544edac28231f79c6e9d14c1d0c302170d8d3844800f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6002186f05dc05c6ae18544edac28231f79c6e9d14c1d0c302170d8d3844800f/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/690e42623d04fa5aa3d49b1cbe7f4ed44923da61e38c096bdb85559d8705aed1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/690e42623d04fa5aa3d49b1cbe7f4ed44923da61e38c096bdb85559d8705aed1/userdata/shm major:0 minor:373 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ea9e8c1f04b4aab4006eeac81a30099666f780462a003dcf5da20294f0e0c9a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ea9e8c1f04b4aab4006eeac81a30099666f780462a003dcf5da20294f0e0c9a/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/84137200c098651df0db031ffd4f9cda174e1d803d3774c76934ca174437c773/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/84137200c098651df0db031ffd4f9cda174e1d803d3774c76934ca174437c773/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ca958337e09414cec8117b907118ca7d199b2255b08746ec08bc6182ccdeebd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ca958337e09414cec8117b907118ca7d199b2255b08746ec08bc6182ccdeebd/userdata/shm major:0 minor:327 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a1182eba2e295734048a99282d575c2778a1cf825dcce39ca7ad71baf84623de/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a1182eba2e295734048a99282d575c2778a1cf825dcce39ca7ad71baf84623de/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b4338a767aa48da2c47aa03999ce26022ebcff53ee47f2cc3b29218644d0b581/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b4338a767aa48da2c47aa03999ce26022ebcff53ee47f2cc3b29218644d0b581/userdata/shm major:0 minor:260 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b8fc48b29d7b468edaf2b1f8982de2890976ae02d3e5e3d7a4613957c394f5a0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b8fc48b29d7b468edaf2b1f8982de2890976ae02d3e5e3d7a4613957c394f5a0/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c18c7b4723c6ab8f8f18e63600604f43a91a32b392ce37c0254c28b41b6e81dd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c18c7b4723c6ab8f8f18e63600604f43a91a32b392ce37c0254c28b41b6e81dd/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e5bd2e832314848947277f81dc2403220f6b50e84660c42dd50d4371840a7200/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e5bd2e832314848947277f81dc2403220f6b50e84660c42dd50d4371840a7200/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e668ee624520a4afd64de9149efe141650aa4ccf915b34e76448afef488ed842/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e668ee624520a4afd64de9149efe141650aa4ccf915b34e76448afef488ed842/userdata/shm major:0 minor:410 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f4ef14f278f6ff49286a2be511bd4cd87dff2ceacccefce6ff649844e972b274/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f4ef14f278f6ff49286a2be511bd4cd87dff2ceacccefce6ff649844e972b274/userdata/shm major:0 minor:368 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06f6f89f-b8f9-45ec-b50a-f3407bfe7ef8/volumes/kubernetes.io~projected/kube-api-access-b8t9t:{mountpoint:/var/lib/kubelet/pods/06f6f89f-b8f9-45ec-b50a-f3407bfe7ef8/volumes/kubernetes.io~projected/kube-api-access-b8t9t major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/071262de-534b-4d01-bc20-784db9852f69/volumes/kubernetes.io~projected/kube-api-access-7rn6f:{mountpoint:/var/lib/kubelet/pods/071262de-534b-4d01-bc20-784db9852f69/volumes/kubernetes.io~projected/kube-api-access-7rn6f major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/071262de-534b-4d01-bc20-784db9852f69/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/071262de-534b-4d01-bc20-784db9852f69/volumes/kubernetes.io~secret/serving-cert major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ea8d3ef-20e9-4ecb-9e50-2df15d7da726/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/0ea8d3ef-20e9-4ecb-9e50-2df15d7da726/volumes/kubernetes.io~secret/agent-certs major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12ce41df-43ef-4b00-be8e-5d3fad591eea/volumes/kubernetes.io~projected/kube-api-access-gsmhx:{mountpoint:/var/lib/kubelet/pods/12ce41df-43ef-4b00-be8e-5d3fad591eea/volumes/kubernetes.io~projected/kube-api-access-gsmhx major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13a7b3bb-060c-4383-a864-0575834b221e/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/13a7b3bb-060c-4383-a864-0575834b221e/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13a7b3bb-060c-4383-a864-0575834b221e/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/13a7b3bb-060c-4383-a864-0575834b221e/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13a7b3bb-060c-4383-a864-0575834b221e/volumes/kubernetes.io~projected/kube-api-access-ht6kh:{mountpoint:/var/lib/kubelet/pods/13a7b3bb-060c-4383-a864-0575834b221e/volumes/kubernetes.io~projected/kube-api-access-ht6kh major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1dccc35b-6206-47f3-b901-bafc5f17b8ad/volumes/kubernetes.io~projected/kube-api-access-s5bg9:{mountpoint:/var/lib/kubelet/pods/1dccc35b-6206-47f3-b901-bafc5f17b8ad/volumes/kubernetes.io~projected/kube-api-access-s5bg9 major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1dccc35b-6206-47f3-b901-bafc5f17b8ad/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/1dccc35b-6206-47f3-b901-bafc5f17b8ad/volumes/kubernetes.io~secret/default-certificate major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1dccc35b-6206-47f3-b901-bafc5f17b8ad/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/1dccc35b-6206-47f3-b901-bafc5f17b8ad/volumes/kubernetes.io~secret/metrics-certs major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1dccc35b-6206-47f3-b901-bafc5f17b8ad/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/1dccc35b-6206-47f3-b901-bafc5f17b8ad/volumes/kubernetes.io~secret/stats-auth major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/215a42f8-881d-4cc4-b670-ea9f4aefb80b/volumes/kubernetes.io~projected/kube-api-access-nwmn4:{mountpoint:/var/lib/kubelet/pods/215a42f8-881d-4cc4-b670-ea9f4aefb80b/volumes/kubernetes.io~projected/kube-api-access-nwmn4 major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/215a42f8-881d-4cc4-b670-ea9f4aefb80b/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/215a42f8-881d-4cc4-b670-ea9f4aefb80b/volumes/kubernetes.io~secret/cert major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26b7328d-dacf-44c2-995b-306e3ef9cf31/volumes/kubernetes.io~projected/kube-api-access-4zplm:{mountpoint:/var/lib/kubelet/pods/26b7328d-dacf-44c2-995b-306e3ef9cf31/volumes/kubernetes.io~projected/kube-api-access-4zplm major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d578e40-60d8-4ebd-b0b5-2f5c274c7903/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/2d578e40-60d8-4ebd-b0b5-2f5c274c7903/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d578e40-60d8-4ebd-b0b5-2f5c274c7903/volumes/kubernetes.io~projected/kube-api-access-gh6h5:{mountpoint:/var/lib/kubelet/pods/2d578e40-60d8-4ebd-b0b5-2f5c274c7903/volumes/kubernetes.io~projected/kube-api-access-gh6h5 major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d578e40-60d8-4ebd-b0b5-2f5c274c7903/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/2d578e40-60d8-4ebd-b0b5-2f5c274c7903/volumes/kubernetes.io~projected/registry-tls major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d578e40-60d8-4ebd-b0b5-2f5c274c7903/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/2d578e40-60d8-4ebd-b0b5-2f5c274c7903/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d578e40-60d8-4ebd-b0b5-2f5c274c7903/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/2d578e40-60d8-4ebd-b0b5-2f5c274c7903/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc/volumes/kubernetes.io~projected/kube-api-access-t6cmc:{mountpoint:/var/lib/kubelet/pods/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc/volumes/kubernetes.io~projected/kube-api-access-t6cmc major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc/volumes/kubernetes.io~secret/samples-operator-tls:{mountpoint:/var/lib/kubelet/pods/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc/volumes/kubernetes.io~secret/samples-operator-tls major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/417d26e1-2a9d-41e3-8940-69626662a2f3/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/417d26e1-2a9d-41e3-8940-69626662a2f3/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:409 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/457675c6-c725-44e5-a857-97aa4c17e655/volumes/kubernetes.io~projected/kube-api-access-9gzjx:{mountpoint:/var/lib/kubelet/pods/457675c6-c725-44e5-a857-97aa4c17e655/volumes/kubernetes.io~projected/kube-api-access-9gzjx major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/472c7524-6a70-420c-aa56-7f3028a53d52/volumes/kubernetes.io~projected/kube-api-access-l9bfd:{mountpoint:/var/lib/kubelet/pods/472c7524-6a70-420c-aa56-7f3028a53d52/volumes/kubernetes.io~projected/kube-api-access-l9bfd major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53ba063f-0db3-4f82-b280-40718c28b479/volumes/kubernetes.io~projected/kube-api-access-jbvvt:{mountpoint:/var/lib/kubelet/pods/53ba063f-0db3-4f82-b280-40718c28b479/volumes/kubernetes.io~projected/kube-api-access-jbvvt major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53ba063f-0db3-4f82-b280-40718c28b479/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls:{mountpoint:/var/lib/kubelet/pods/53ba063f-0db3-4f82-b280-40718c28b479/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/566b2dae-dd3e-4c63-b4d2-e30667a5fd07/volumes/kubernetes.io~projected/kube-api-access-cqn4t:{mountpoint:/var/lib/kubelet/pods/566b2dae-dd3e-4c63-b4d2-e30667a5fd07/volumes/kubernetes.io~projected/kube-api-access-cqn4t major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72ef59a1-3734-47b7-9074-f683ca94dd32/volumes/kubernetes.io~projected/kube-api-access-qczmw:{mountpoint:/var/lib/kubelet/pods/72ef59a1-3734-47b7-9074-f683ca94dd32/volumes/kubernetes.io~projected/kube-api-access-qczmw major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72ef59a1-3734-47b7-9074-f683ca94dd32/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/72ef59a1-3734-47b7-9074-f683ca94dd32/volumes/kubernetes.io~secret/metrics-tls major:0 minor:376 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7321845c-9a95-415b-b6db-2bf983d77fa5/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/7321845c-9a95-415b-b6db-2bf983d77fa5/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7321845c-9a95-415b-b6db-2bf983d77fa5/volumes/kubernetes.io~projected/kube-api-access-2t56h:{mountpoint:/var/lib/kubelet/pods/7321845c-9a95-415b-b6db-2bf983d77fa5/volumes/kubernetes.io~projected/kube-api-access-2t56h major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7321845c-9a95-415b-b6db-2bf983d77fa5/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/7321845c-9a95-415b-b6db-2bf983d77fa5/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82b3b5e2-09f1-49c0-8d72-c85187918223/volumes/kubernetes.io~projected/kube-api-access-64dcq:{mountpoint:/var/lib/kubelet/pods/82b3b5e2-09f1-49c0-8d72-c85187918223/volumes/kubernetes.io~projected/kube-api-access-64dcq major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82b3b5e2-09f1-49c0-8d72-c85187918223/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/82b3b5e2-09f1-49c0-8d72-c85187918223/volumes/kubernetes.io~secret/metrics-certs major:0 minor:408 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/volumes/kubernetes.io~projected/kube-api-access-bfz5j:{mountpoint:/var/lib/kubelet/pods/a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/volumes/kubernetes.io~projected/kube-api-access-bfz5j major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8d56c63-207c-42b3-8946-c993d0fde66b/volumes/kubernetes.io~projected/kube-api-access-tnf2q:{mountpoint:/var/lib/kubelet/pods/a8d56c63-207c-42b3-8946-c993d0fde66b/volumes/kubernetes.io~projected/kube-api-access-tnf2q major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae96a9c1-10a6-4050-bca0-6a2e7859d3c3/volumes/kubernetes.io~projected/kube-api-access-9qw9l:{mountpoint:/var/lib/kubelet/pods/ae96a9c1-10a6-4050-bca0-6a2e7859d3c3/volumes/kubernetes.io~projected/kube-api-access-9qw9l major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae96a9c1-10a6-4050-bca0-6a2e7859d3c3/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/ae96a9c1-10a6-4050-bca0-6a2e7859d3c3/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca5741f4-603c-448e-9088-0543eef08544/volumes/kubernetes.io~projected/kube-api-access-bjlw7:{mountpoint:/var/lib/kubelet/pods/ca5741f4-603c-448e-9088-0543eef08544/volumes/kubernetes.io~projected/kube-api-access-bjlw7 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb204d6b-3758-4db4-bd68-0eacec87b805/volumes/kubernetes.io~projected/kube-api-access-njqr9:{mountpoint:/var/lib/kubelet/pods/cb204d6b-3758-4db4-bd68-0eacec87b805/volumes/kubernetes.io~projected/kube-api-access-njqr9 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542/volumes/kubernetes.io~secret/networking-console-plugin-cert:{mountpoint:/var/lib/kubelet/pods/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542/volumes/kubernetes.io~secret/networking-console-plugin-cert major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecdc26b5-be99-4928-8fff-203cc2c19ad6/volumes/kubernetes.io~projected/kube-api-access-jjvmh:{mountpoint:/var/lib/kubelet/pods/ecdc26b5-be99-4928-8fff-203cc2c19ad6/volumes/kubernetes.io~projected/kube-api-access-jjvmh major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee0440bc-1614-4209-8744-6d197a019b5c/volumes/kubernetes.io~projected/kube-api-access-jwnw9:{mountpoint:/var/lib/kubelet/pods/ee0440bc-1614-4209-8744-6d197a019b5c/volumes/kubernetes.io~projected/kube-api-access-jwnw9 major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee0440bc-1614-4209-8744-6d197a019b5c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/ee0440bc-1614-4209-8744-6d197a019b5c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb045f55-e48d-4cac-88cf-90f90c225cbb/volumes/kubernetes.io~projected/kube-api-access-bfpxg:{mountpoint:/var/lib/kubelet/pods/fb045f55-e48d-4cac-88cf-90f90c225cbb/volumes/kubernetes.io~projected/kube-api-access-bfpxg major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c/volumes/kubernetes.io~projected/kube-api-access-zvp6g:{mountpoint:/var/lib/kubelet/pods/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c/volumes/kubernetes.io~projected/kube-api-access-zvp6g major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c/volumes/kubernetes.io~secret/serving-cert major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc560bbe-b31a-48e8-8416-dda72f49dae1/volumes/kubernetes.io~projected/kube-api-access-5t2t2:{mountpoint:/var/lib/kubelet/pods/fc560bbe-b31a-48e8-8416-dda72f49dae1/volumes/kubernetes.io~projected/kube-api-access-5t2t2 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff5ad4d1-4fd9-4374-b11c-78596c9cafa8/volumes/kubernetes.io~projected/kube-api-access-cgmgr:{mountpoint:/var/lib/kubelet/pods/ff5ad4d1-4fd9-4374-b11c-78596c9cafa8/volumes/kubernetes.io~projected/kube-api-access-cgmgr major:0 minor:233 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/3898cf428b62c818501b64afd448697b149f690cb6a1bccd79aa043d024499af/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/c235ded86fe79182aca42716294f858007bfcf0a0f7198c3538521cf2a85795a/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/959c1714e507959ebf515c883d928bbe74cfed7808c15fe6ef319d146e3a5515/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/cb1aa3ef5d882cf746a39266adc7169898e738d7044149b1968b4efdc0e5aad2/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/01dce794528dad946b4e904aeb8572ec5c0b3ca05b734cf950f0ccc97a139392/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/a0cf652453ec72eda20b2eac75ea82a0cab25e9b97a56e5a55043348f58b35eb/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/aefafa4745b45cb6b7dcb11e87d5d45cd97c2d1653fb90e191eee2fa3482cc1e/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/5f6a02d35bbe439ac93cbd27c786d38f9fc3fd880300f46fb1667d69275c66c4/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/5e483412570908adbbe7d89fb8fb5ab6bfb9f841276be884ade02e89334e2eb5/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/5b30c67e7cbe1475c849fe119ee2cd06eb807f24835828748d49da98cb7b9958/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/ab8a927d250dc28efaa86224b402e96553af301acbdbbb0043785cd7561f848a/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/ea5bd03998c8f0e9c0da3c56c60875e9f6123e005856c31cb340fe550faaceec/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/6bf9947bcbc0c354dc9cda6738259e2be8d0f4e2b124361ce5edd5b27eba01aa/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/3970cfc0ba48ce0a60794f30348e6fcccf21d50fd968a87a93f7070483ca399a/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/f4dcc5c97c45e5c927f4dc58f167bd0e52b684705c04abd75e75c1698ba9875d/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/85870f9fcf3e269055000a6e86cc7fd20e71f5772e07fabb3ffb4385f4a75fdd/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/6d688c95a6da85c6a45777d21115646a445bd3bc8ad032f509e6684aac80ce09/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/8602cf5aa791e19f3bccd4a13a3d57c0590b751dd47b66366c8042bbb4fff61e/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/31a4a7047981b82ae6736923d006f84d3ae8fa340f95adf2c85ee1f6d780a131/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-186:{mountpoint:/var/lib/containers/storage/overlay/003d0626333754b2bd6619cb7e7f619c5d281aa756587efa8f3aeb788aaac52d/merged major:0 minor:186 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/cf3c3b4f7c84dbba9244ac79202ea4428a9a1357256633065b026ac77c287ee0/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-224:{mountpoint:/var/lib/containers/storage/overlay/7955a1f7fe065b9b610776e951eef9f6314edec1c7b9a6e1c4d9ea873709a304/merged major:0 minor:224 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/e3b57cd678096d8c8e96b319357675d2c25ce15a2193f2d709bfba9755bba5b6/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/a00f5c93fba71aa37da211668eab5adbc1835775fb425bc66099e81842583eba/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/14b8696aa7761fa0f762c7b7385db3046ebd802f02262dfe2e03e0ea3ac06b56/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/1cef1550a9a35ca682e423b666fe28a55c0cb4f83c4dbf8936d4b00bf6fd9030/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/b0a407078bcf1addb545e07486eed3b3616dc5e187cbe8149803446695601716/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/fb8c743c8437894b1bf99d0c06ebd7cf99d517b162d761561fc693aa5ac637ac/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/dcedb8db95b5cb2953f7cc7541ca3bf3fe5ae0796b6011ee8f69d090b8313cd8/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/96579459ab527647cc006f00fe73a338fae31d3e6a682d6a317b927a0425521f/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/31c7e507e4232d5588be32154d54ebe14e49e273e895a2e5a451d4c1e0a7044e/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/076176db233e8286e758e8503486e259213c3aecc96cee149c78f3a5db3d0870/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/fd22609abc33024e02e04032cfca0e5fd97ba852d7e47b1a9a5abc0dc9edb9f9/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/2c3971a65d7a8849b9a2802f51aa48b1ab4b9f5707836b9010aca601525efe81/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/02cda22c18b36d3761b8e712d67f19210ee2b8583c6d8d969eb8ac341a29836b/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/081b60fef529ed05571ce056054873227da6872ab070b12232cc5c678869d695/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/5770fa6225889524b728feb676640a500d5e39f7f2a81b7b4deb549475109a03/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/b3829f6dbfacdca5933d56b8da00155ec6df86137bc97b9dc99a832186ba00a9/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/b5ab7ca68c37cbd3452d1077e8f3762996391b9fef9078ee777e5036c80d6112/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/dff6fa0d5b4cba79a9d68ead0ac82bda04da0957026929e51eea2b695f9b1567/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/5cc1408407a02b99f5ce762dc2ab3f5da3452c8a3e1e44c2b8f985b9007fe054/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/37ea27945ad55e1c3ca239a312714297e07b89daf89ef0fd2c177565a5b3b557/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/5fd543d07b6cb64af1e6ae5ce6c8ba803075f17e94cfec791f84a84e43984563/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/fc6efe9671c6c373128fe3806eebac7de62feb1420e593e87a64a4e91b965b07/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/a0652fc067e2e81238e7fc75b6b0a5fe3501d35902685629dc0e25cd1518cd58/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/3f2d0a9ffd72bcce6662984e963a6115337e9b270fef1229398ef686de747c71/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/7605bd088f94621cf9a1ddec8857ddff5ad44d09c093049697554806b0ee3519/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/25faca43688c4081dcccd23cf6b03fea03a4af47d7b5604880b141e34bd68131/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/4c694faef7c88626738e0486cc440b336cf4411c62731eb3c45e656d8072e71f/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/999a23a1acc7a470619641fc20db17705bc834082d2d631498da0afabc8a050b/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-384:{mountpoint:/var/lib/containers/storage/overlay/0e8ec3a64b853b621e68cba2d0969b4ab27c7828622e87243484a0df42e165cf/merged major:0 minor:384 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/4fdc24fdff7b045b0be3d2d25e82de1cefb12d557d14edac90c38bfe99cf05d9/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-392:{mountpoint:/var/lib/containers/storage/overlay/f6e927e7a541b3cda508c4ee95e55f7f46734a4158745bb4f6932fdcac56264e/merged major:0 minor:392 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/7de0f51350e0eb39294d19eca45183a92c03ad50e957aae42689522ab19e5f60/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/436183d77eeb0cf942d6ba8627b9a891ae65f3cfc2a27ad1d9730efcf2d8da7b/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-402:{mountpoint:/var/lib/containers/storage/overlay/b2cfc82d68ea6b5d752cb656144babf3f1ea63917963c0dc5671be22b6989a3e/merged major:0 minor:402 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/24201f17fdd3f03956e60e4d410176e8ec416e53c97ea5f6656872ab35242a07/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-416:{mountpoint:/var/lib/containers/storage/overlay/f2d7a3417a29d337c309babd272e82a53b171aed574ad99b1ce180949b5f38e4/merged major:0 minor:416 fsType:overlay blockSize:0} overlay_0-418:{mountpoint:/var/lib/containers/storage/overlay/ee26787032577a5ebdd35061e8f5e1373821acec7b1180887529f4a1a81f0205/merged major:0 minor:418 fsType:overlay blockSize:0} overlay_0-420:{mountpoint:/var/lib/containers/storage/overlay/b8dcd126b44922dc6bd90a9b4d5ae6bb7b40fca848e89e803823b86ba2642fdb/merged major:0 minor:420 fsType:overlay blockSize:0} overlay_0-426:{mountpoint:/var/lib/containers/storage/overlay/6e2db2c1681c70716e0d940a5146baaf6ced69abc58020fc67d0d9dbae727708/merged major:0 minor:426 fsType:overlay blockSize:0} overlay_0-432:{mountpoint:/var/lib/containers/storage/overlay/1b3e292ffc2d397fa73cc3ed90c782d345039d6ffcfc5fc66356da7edf35ea9d/merged major:0 minor:432 fsType:overlay blockSize:0} overlay_0-434:{mountpoint:/var/lib/containers/storage/overlay/27bb0848e6378380f1d5b704aeb84d29a7daba33a0ea3ec8b26058ed1a4df549/merged major:0 minor:434 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/391ad644638bedfdf55f7feba0460020e62eecb234874ce8b04a222a8725de9f/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-438:{mountpoint:/var/lib/containers/storage/overlay/ba5b01b194a4195ae6636b3303caa2afee71d43d3b46fb1714ae1cba06e02cce/merged major:0 minor:438 fsType:overlay blockSize:0} overlay_0-444:{mountpoint:/var/lib/containers/storage/overlay/2114ff42fd062be783abdb1f13991ebfc228edc76973201954be77d29e7ff6e4/merged major:0 minor:444 fsType:overlay blockSize:0} overlay_0-456:{mountpoint:/var/lib/containers/storage/overlay/7528c2673463ab4335700be4348b4045f24f03c6228f77cb728fb4ccdb7883de/merged major:0 minor:456 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/62ab1dab1606ecb5efca8234ab13c6f33b0ed0ed8e32eea190b3ef238f966a68/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/5b1c84758d5288deef5180d969f9fcbf5ac7ca97e23e337e353c36f9f9f32e03/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/9f61ee32f95bc5025cba46882c77a2d4a31420006f837a83588e3b2432724b14/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/398abc126fbddbd657e576b8d41683b9765875d52d72aa8463aa91578b79e2d3/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/170cbb4c427a6292b6e7f522acaeabb16ecfd73032b5b027f37e550686f935e3/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/2c73148fc699c68e943643c52c3b4111004e7a486f88a87a73a78f007d57b0a7/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/fb044238daab184df74beae285afe882834278daf4aaea6428d8423997d36a9f/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/4ac46e1be60e82b0c5ee2129ef1113654fbce1854e11066ae3c1004d4aba7284/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/a139d6d027ea0d35def60cd6944a54a93925531716f49093387ca6994886ff1b/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/6e46183663a4a425d8d27a4947465bce456251316682ef92428539e97a9ffa90/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 22:48:33.823526 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.823010 5727 manager.go:217] Machine: {Timestamp:2026-11-02 22:48:33.822196407 +0000 UTC m=+0.060694061 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101244 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f25bc58672d08509426113e404a38 SystemUUID:ec2f25bc-5867-2d08-5094-26113e404a38 BootID:f3231f38-0bcb-4949-9784-5daedbcd92ab Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/071262de-534b-4d01-bc20-784db9852f69/volumes/kubernetes.io~projected/kube-api-access-7rn6f DeviceMajor:0 DeviceMinor:234 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc/volumes/kubernetes.io~secret/samples-operator-tls DeviceMajor:0 DeviceMinor:372 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/0ea8d3ef-20e9-4ecb-9e50-2df15d7da726/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:50 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/2d578e40-60d8-4ebd-b0b5-2f5c274c7903/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:221 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2393b59785260194d19a54699f33e597481d2bf7ebd4c3b1679768477af03b4e/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/13a7b3bb-060c-4383-a864-0575834b221e/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/cb204d6b-3758-4db4-bd68-0eacec87b805/volumes/kubernetes.io~projected/kube-api-access-njqr9 DeviceMajor:0 DeviceMinor:52 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/fc560bbe-b31a-48e8-8416-dda72f49dae1/volumes/kubernetes.io~projected/kube-api-access-5t2t2 DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc/volumes/kubernetes.io~projected/kube-api-access-t6cmc DeviceMajor:0 DeviceMinor:230 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/2d578e40-60d8-4ebd-b0b5-2f5c274c7903/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:232 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/215a42f8-881d-4cc4-b670-ea9f4aefb80b/volumes/kubernetes.io~projected/kube-api-access-nwmn4 DeviceMajor:0 DeviceMinor:241 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-186 DeviceMajor:0 DeviceMinor:186 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1dccc35b-6206-47f3-b901-bafc5f17b8ad/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:211 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/72ef59a1-3734-47b7-9074-f683ca94dd32/volumes/kubernetes.io~projected/kube-api-access-qczmw DeviceMajor:0 DeviceMinor:240 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ee0440bc-1614-4209-8744-6d197a019b5c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:341 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7321845c-9a95-415b-b6db-2bf983d77fa5/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/320f557ca944b584051436e7f2df20c9bdd300bcbb801c8a193fc912ffec0505/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb045f55-e48d-4cac-88cf-90f90c225cbb/volumes/kubernetes.io~projected/kube-api-access-bfpxg DeviceMajor:0 DeviceMinor:54 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c18c7b4723c6ab8f8f18e63600604f43a91a32b392ce37c0254c28b41b6e81dd/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e5bd2e832314848947277f81dc2403220f6b50e84660c42dd50d4371840a7200/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b8fc48b29d7b468edaf2b1f8982de2890976ae02d3e5e3d7a4613957c394f5a0/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-392 DeviceMajor:0 DeviceMinor:392 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a8d56c63-207c-42b3-8946-c993d0fde66b/volumes/kubernetes.io~projected/kube-api-access-tnf2q DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/12d0b5f3c4797a1b4b8958ae37822899ad7f836fe7527324eff473eb7eee945b/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/20aeb9b1286b40414ad4b557e23c6578bd0da566586adfb27316b45cd1632aef/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6002186f05dc05c6ae18544edac28231f79c6e9d14c1d0c302170d8d3844800f/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b4338a767aa48da2c47aa03999ce26022ebcff53ee47f2cc3b29218644d0b581/userdata/shm DeviceMajor:0 DeviceMinor:260 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31b84ffae7960055d9afb0250f2959a11bbbc6bd0d76c87a2c5a2c9021b420e0/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-416 DeviceMajor:0 DeviceMinor:416 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/26b7328d-dacf-44c2-995b-306e3ef9cf31/volumes/kubernetes.io~projected/kube-api-access-4zplm DeviceMajor:0 DeviceMinor:53 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/82b3b5e2-09f1-49c0-8d72-c85187918223/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:408 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e668ee624520a4afd64de9149efe141650aa4ccf915b34e76448afef488ed842/userdata/shm DeviceMajor:0 DeviceMinor:410 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-438 DeviceMajor:0 DeviceMinor:438 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/13a7b3bb-060c-4383-a864-0575834b221e/volumes/kubernetes.io~projected/kube-api-access-ht6kh DeviceMajor:0 DeviceMinor:51 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ecdc26b5-be99-4928-8fff-203cc2c19ad6/volumes/kubernetes.io~projected/kube-api-access-jjvmh DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/12ce41df-43ef-4b00-be8e-5d3fad591eea/volumes/kubernetes.io~projected/kube-api-access-gsmhx DeviceMajor:0 DeviceMinor:202 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e20810858ab32fdce1dd75df90c3dbfc2084c6aca8fa0711e884db900522380/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-434 DeviceMajor:0 DeviceMinor:434 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/volumes/kubernetes.io~projected/kube-api-access-bfz5j DeviceMajor:0 DeviceMinor:215 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c/volumes/kubernetes.io~projected/kube-api-access-zvp6g DeviceMajor:0 DeviceMinor:218 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/06f6f89f-b8f9-45ec-b50a-f3407bfe7ef8/volumes/kubernetes.io~projected/kube-api-access-b8t9t DeviceMajor:0 DeviceMinor:326 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2035e8f997a9e7d0935f37f90194f850ae3c24b21829a8b22471026bfae92ce8/userdata/shm DeviceMajor:0 DeviceMinor:379 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-402 DeviceMajor:0 DeviceMinor:402 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5399dea54dbec22dce782d3b4c70b29d1f0dfd0191efc4025948b94dc5a9cbd9/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/53ba063f-0db3-4f82-b280-40718c28b479/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls DeviceMajor:0 DeviceMinor:363 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7321845c-9a95-415b-b6db-2bf983d77fa5/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/1dccc35b-6206-47f3-b901-bafc5f17b8ad/volumes/kubernetes.io~projected/kube-api-access-s5bg9 DeviceMajor:0 DeviceMinor:217 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ee0440bc-1614-4209-8744-6d197a019b5c/volumes/kubernetes.io~projected/kube-api-access-jwnw9 DeviceMajor:0 DeviceMinor:309 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ca5741f4-603c-448e-9088-0543eef08544/volumes/kubernetes.io~projected/kube-api-access-bjlw7 DeviceMajor:0 DeviceMinor:216 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2762b05b8165a3da012d5281faadff367ecfd35437f6345d467e82f78916402a/userdata/shm DeviceMajor:0 DeviceMinor:377 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4fba275614bd2ddaf44e9966d406d568f3b834031d711e5499040da2ca9c527d/userdata/shm DeviceMajor:0 DeviceMinor:381 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-444 DeviceMajor:0 DeviceMinor:444 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/278cf9de5cdac5e81c0f35ddc4d882c8f4fd7f0a7a2cb7f4717d54c016007ec0/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4a29976e47440460e64c2e170cbc823f58c5efb479a898fb03278815767a5c2b/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/2d578e40-60d8-4ebd-b0b5-2f5c274c7903/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:370 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-456 DeviceMajor:0 DeviceMinor:456 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ea9e8c1f04b4aab4006eeac81a30099666f780462a003dcf5da20294f0e0c9a/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/53ba063f-0db3-4f82-b280-40718c28b479/volumes/kubernetes.io~projected/kube-api-access-jbvvt DeviceMajor:0 DeviceMinor:203 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/071262de-534b-4d01-bc20-784db9852f69/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:222 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/13a7b3bb-060c-4383-a864-0575834b221e/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/149bfac297d1959995d6a77e8d0cb2e89a62a5efe665a352f8dd48c6a210de40/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ae96a9c1-10a6-4050-bca0-6a2e7859d3c3/volumes/kubernetes.io~projected/kube-api-access-9qw9l DeviceMajor:0 DeviceMinor:204 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/1dccc35b-6206-47f3-b901-bafc5f17b8ad/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:212 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f4ef14f278f6ff49286a2be511bd4cd87dff2ceacccefce6ff649844e972b274/userdata/shm DeviceMajor:0 DeviceMinor:368 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2d578e40-60d8-4ebd-b0b5-2f5c274c7903/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:225 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-418 DeviceMajor:0 DeviceMinor:418 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/566b2dae-dd3e-4c63-b4d2-e30667a5fd07/volumes/kubernetes.io~projected/kube-api-access-cqn4t DeviceMajor:0 DeviceMinor:268 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7321845c-9a95-415b-b6db-2bf983d77fa5/volumes/kubernetes.io~projected/kube-api-access-2t56h DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a1182eba2e295734048a99282d575c2778a1cf825dcce39ca7ad71baf84623de/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ff5ad4d1-4fd9-4374-b11c-78596c9cafa8/volumes/kubernetes.io~projected/kube-api-access-cgmgr DeviceMajor:0 DeviceMinor:233 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/82b3b5e2-09f1-49c0-8d72-c85187918223/volumes/kubernetes.io~projected/kube-api-access-64dcq DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/57a51cd7c0976e7981c9f7af8100910430f2502147ec3d8f927b084372b98d6f/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-384 DeviceMajor:0 DeviceMinor:384 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-420 DeviceMajor:0 DeviceMinor:420 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-432 DeviceMajor:0 DeviceMinor:432 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5ed5ea0522eca41e9753ad49cca68438d6ba6ae49d49745bcc1487e938928f84/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/472c7524-6a70-420c-aa56-7f3028a53d52/volumes/kubernetes.io~projected/kube-api-access-l9bfd DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d0f15b28f79d9d966377869a930b9cd17054031a166bfbbd0473f25097a1d2a/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2f5f80417d312f9989700350646e7b60429b5eb784cc4e0e275da19371dbc8d3/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1dbff91d6a416a7a712a6415f02a9e2f4fb91cbbeeacad097a2516b6781d2736/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542/volumes/kubernetes.io~secret/networking-console-plugin-cert DeviceMajor:0 DeviceMinor:371 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/061fd09ec7499698a85f2e9bb065f4baa4da0ef59ed108bd060b11ee0c4b7d9e/userdata/shm DeviceMajor:0 DeviceMinor:412 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/457675c6-c725-44e5-a857-97aa4c17e655/volumes/kubernetes.io~projected/kube-api-access-9gzjx DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/215a42f8-881d-4cc4-b670-ea9f4aefb80b/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:239 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/72ef59a1-3734-47b7-9074-f683ca94dd32/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:376 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-224 DeviceMajor:0 DeviceMinor:224 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/417d26e1-2a9d-41e3-8940-69626662a2f3/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:409 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/84137200c098651df0db031ffd4f9cda174e1d803d3774c76934ca174437c773/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ae96a9c1-10a6-4050-bca0-6a2e7859d3c3/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:210 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/2d578e40-60d8-4ebd-b0b5-2f5c274c7903/volumes/kubernetes.io~projected/kube-api-access-gh6h5 DeviceMajor:0 DeviceMinor:236 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/690e42623d04fa5aa3d49b1cbe7f4ed44923da61e38c096bdb85559d8705aed1/userdata/shm DeviceMajor:0 DeviceMinor:373 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-426 DeviceMajor:0 DeviceMinor:426 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ca958337e09414cec8117b907118ca7d199b2255b08746ec08bc6182ccdeebd/userdata/shm DeviceMajor:0 DeviceMinor:327 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/1dccc35b-6206-47f3-b901-bafc5f17b8ad/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:367 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:061fd09ec749969 MacAddress:62:36:09:4a:02:90 Speed:10000 Mtu:8901} {Name:1dbff91d6a416a7 MacAddress:1a:8f:66:08:53:7f Speed:10000 Mtu:8901} {Name:2035e8f997a9e7d MacAddress:b2:46:7a:b0:03:5f Speed:10000 Mtu:8901} {Name:2393b5978526019 MacAddress:82:47:bf:f1:c1:5e Speed:10000 Mtu:8901} {Name:2762b05b8165a3d MacAddress:ba:ea:9f:4b:4c:d7 Speed:10000 Mtu:8901} {Name:2f5f80417d312f9 MacAddress:a2:7e:2d:46:02:16 Speed:10000 Mtu:8901} {Name:31b84ffae796005 MacAddress:a6:86:6e:4d:4f:6e Speed:10000 Mtu:8901} {Name:320f557ca944b58 MacAddress:ea:52:2a:c6:26:f4 Speed:10000 Mtu:8901} {Name:4a29976e4744046 MacAddress:52:b3:b1:30:a3:46 Speed:10000 Mtu:8901} {Name:4e20810858ab32f MacAddress:56:23:f1:b9:5d:c8 Speed:10000 Mtu:8901} {Name:4fba275614bd2dd MacAddress:e6:a4:5d:f0:c5:7e Speed:10000 Mtu:8901} {Name:6002186f05dc05c MacAddress:2e:ba:d2:06:e0:ae Speed:10000 Mtu:8901} {Name:690e42623d04fa5 MacAddress:ee:7e:30:07:ae:96 Speed:10000 Mtu:8901} {Name:6ea9e8c1f04b4aa MacAddress:7a:c7:1d:d2:8e:4f Speed:10000 Mtu:8901} {Name:9ca958337e09414 MacAddress:c2:0e:97:00:60:63 Speed:10000 Mtu:8901} {Name:a1182eba2e29573 MacAddress:46:eb:e5:4a:4a:27 Speed:10000 Mtu:8901} {Name:b4338a767aa48da MacAddress:d6:1f:b5:33:32:d0 Speed:10000 Mtu:8901} {Name:b8fc48b29d7b468 MacAddress:ce:8f:4e:00:a6:a3 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:47:3c:9d:e1:f9 Speed:0 Mtu:9001} {Name:br-int MacAddress:be:ee:94:07:5e:30 Speed:0 Mtu:8901} {Name:e5bd2e832314848 MacAddress:02:f3:86:27:83:e6 Speed:10000 Mtu:8901} {Name:e668ee624520a4a MacAddress:86:d0:fe:d6:bd:b9 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:47:3c:9d:e1:f9 Speed:0 Mtu:9001} {Name:f4ef14f278f6ff4 MacAddress:ae:4b:a5:69:81:46 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:f2:46:14:8e:11:8b Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:76:68:e3:b2:9c:15 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 22:48:33.823526 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.823530 5727 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 22:48:33.823934 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.823579 5727 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 22:48:33.824007 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.823975 5727 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 22:48:33.824191 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.824011 5727 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-176.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 22:48:33.824282 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.824194 5727 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 22:48:33.824282 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.824207 5727 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 22:48:33.824282 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.824221 5727 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 22:48:33.824282 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.824249 5727 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 22:48:33.824282 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.824276 5727 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 22:48:33.824510 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.824489 5727 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 22:48:33.824627 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.824615 5727 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 22:48:33.824687 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.824662 5727 kubelet.go:493] "Attempting to sync node with API server" Nov 02 22:48:33.824687 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.824681 5727 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 22:48:33.824774 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.824695 5727 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 22:48:33.824774 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.824708 5727 kubelet.go:405] "Adding apiserver pod source" Nov 02 22:48:33.824774 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.824720 5727 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 22:48:33.825444 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.825427 5727 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 22:48:33.825444 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.825443 5727 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 22:48:33.825961 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.825943 5727 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 22:48:33.828737 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.828722 5727 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 22:48:33.828737 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.828736 5727 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 22:48:33.828857 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.828847 5727 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 22:48:33.829045 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.829034 5727 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 22:48:33.829075 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.829064 5727 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 22:48:33.829200 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.829190 5727 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 22:48:33.829232 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.829205 5727 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 22:48:33.829232 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.829212 5727 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 22:48:33.829232 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.829217 5727 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 22:48:33.829232 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.829222 5727 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 22:48:33.829351 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.829233 5727 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 22:48:33.829351 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.829243 5727 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 22:48:33.829351 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.829251 5727 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 22:48:33.829351 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.829259 5727 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 22:48:33.829351 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.829265 5727 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 22:48:33.829351 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.829293 5727 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 22:48:33.829351 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.829308 5727 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 22:48:33.829351 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.829336 5727 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 22:48:33.829351 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.829344 5727 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 22:48:33.832962 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.832949 5727 server.go:1263] "Started kubelet" Nov 02 22:48:33.833450 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.833084 5727 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 22:48:33.833450 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.833173 5727 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 22:48:33.833608 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.833576 5727 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 22:48:33.833634 ip-10-0-135-176 systemd[1]: Started Kubernetes Kubelet. Nov 02 22:48:33.833741 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.833028 5727 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 22:48:33.834942 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.834925 5727 server.go:317] "Adding debug handlers to kubelet server" Nov 02 22:48:33.841837 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.841817 5727 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 22:48:33.841941 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.841836 5727 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 22:48:33.841941 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.841900 5727 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 22:42:23 +0000 UTC" deadline="2028-04-16 06:20:11.739326623 +0000 UTC" Nov 02 22:48:33.841941 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.841935 5727 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12727h31m37.897394764s" Nov 02 22:48:33.842175 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.841948 5727 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 22:48:33.842175 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.841960 5727 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 22:48:33.842175 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:48:33.841977 5727 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-176.us-west-1.compute.internal\" not found" Nov 02 22:48:33.842175 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.842047 5727 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 22:48:33.842625 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.842603 5727 factory.go:55] Registering systemd factory Nov 02 22:48:33.842704 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.842629 5727 factory.go:223] Registration of the systemd container factory successfully Nov 02 22:48:33.842839 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.842825 5727 factory.go:153] Registering CRI-O factory Nov 02 22:48:33.842899 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.842843 5727 factory.go:223] Registration of the crio container factory successfully Nov 02 22:48:33.842952 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.842921 5727 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 22:48:33.843013 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.842952 5727 factory.go:103] Registering Raw factory Nov 02 22:48:33.843013 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.842993 5727 manager.go:1199] Started watching for new ooms in manager Nov 02 22:48:33.843517 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.843365 5727 manager.go:319] Starting recovery of all containers Nov 02 22:48:33.845096 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845064 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d578e40-60d8-4ebd-b0b5-2f5c274c7903" volumeName="kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-tls" seLinuxMountContext="" Nov 02 22:48:33.845096 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845094 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72ef59a1-3734-47b7-9074-f683ca94dd32" volumeName="kubernetes.io/secret/72ef59a1-3734-47b7-9074-f683ca94dd32-metrics-tls" seLinuxMountContext="" Nov 02 22:48:33.845260 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845105 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5" volumeName="kubernetes.io/configmap/a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5-config" seLinuxMountContext="" Nov 02 22:48:33.845260 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845113 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5" volumeName="kubernetes.io/projected/a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5-kube-api-access-bfz5j" seLinuxMountContext="" Nov 02 22:48:33.845260 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845140 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542" volumeName="kubernetes.io/configmap/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542-nginx-conf" seLinuxMountContext="" Nov 02 22:48:33.845260 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845150 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12ce41df-43ef-4b00-be8e-5d3fad591eea" volumeName="kubernetes.io/projected/12ce41df-43ef-4b00-be8e-5d3fad591eea-kube-api-access-gsmhx" seLinuxMountContext="" Nov 02 22:48:33.845260 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845157 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="417d26e1-2a9d-41e3-8940-69626662a2f3" volumeName="kubernetes.io/secret/417d26e1-2a9d-41e3-8940-69626662a2f3-original-pull-secret" seLinuxMountContext="" Nov 02 22:48:33.845260 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845164 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53ba063f-0db3-4f82-b280-40718c28b479" volumeName="kubernetes.io/projected/53ba063f-0db3-4f82-b280-40718c28b479-kube-api-access-jbvvt" seLinuxMountContext="" Nov 02 22:48:33.845260 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845175 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53ba063f-0db3-4f82-b280-40718c28b479" volumeName="kubernetes.io/secret/53ba063f-0db3-4f82-b280-40718c28b479-cluster-monitoring-operator-tls" seLinuxMountContext="" Nov 02 22:48:33.845260 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845183 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72ef59a1-3734-47b7-9074-f683ca94dd32" volumeName="kubernetes.io/projected/72ef59a1-3734-47b7-9074-f683ca94dd32-kube-api-access-qczmw" seLinuxMountContext="" Nov 02 22:48:33.845260 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845192 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7321845c-9a95-415b-b6db-2bf983d77fa5" volumeName="kubernetes.io/configmap/7321845c-9a95-415b-b6db-2bf983d77fa5-env-overrides" seLinuxMountContext="" Nov 02 22:48:33.845260 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845199 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae96a9c1-10a6-4050-bca0-6a2e7859d3c3" volumeName="kubernetes.io/projected/ae96a9c1-10a6-4050-bca0-6a2e7859d3c3-kube-api-access-9qw9l" seLinuxMountContext="" Nov 02 22:48:33.845260 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845206 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c" volumeName="kubernetes.io/secret/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-serving-cert" seLinuxMountContext="" Nov 02 22:48:33.845260 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845213 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d578e40-60d8-4ebd-b0b5-2f5c274c7903" volumeName="kubernetes.io/secret/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-installation-pull-secrets" seLinuxMountContext="" Nov 02 22:48:33.845260 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845224 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ea8d3ef-20e9-4ecb-9e50-2df15d7da726" volumeName="kubernetes.io/configmap/0ea8d3ef-20e9-4ecb-9e50-2df15d7da726-konnectivity-ca" seLinuxMountContext="" Nov 02 22:48:33.845260 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845231 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72ef59a1-3734-47b7-9074-f683ca94dd32" volumeName="kubernetes.io/empty-dir/72ef59a1-3734-47b7-9074-f683ca94dd32-tmp-dir" seLinuxMountContext="" Nov 02 22:48:33.845260 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845238 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c" volumeName="kubernetes.io/empty-dir/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-snapshots" seLinuxMountContext="" Nov 02 22:48:33.845260 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845245 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="071262de-534b-4d01-bc20-784db9852f69" volumeName="kubernetes.io/projected/071262de-534b-4d01-bc20-784db9852f69-kube-api-access-7rn6f" seLinuxMountContext="" Nov 02 22:48:33.845260 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845253 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1dccc35b-6206-47f3-b901-bafc5f17b8ad" volumeName="kubernetes.io/projected/1dccc35b-6206-47f3-b901-bafc5f17b8ad-kube-api-access-s5bg9" seLinuxMountContext="" Nov 02 22:48:33.845260 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845260 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc" volumeName="kubernetes.io/secret/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-samples-operator-tls" seLinuxMountContext="" Nov 02 22:48:33.845260 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845267 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee0440bc-1614-4209-8744-6d197a019b5c" volumeName="kubernetes.io/projected/ee0440bc-1614-4209-8744-6d197a019b5c-kube-api-access-jwnw9" seLinuxMountContext="" Nov 02 22:48:33.845260 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845274 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="215a42f8-881d-4cc4-b670-ea9f4aefb80b" volumeName="kubernetes.io/projected/215a42f8-881d-4cc4-b670-ea9f4aefb80b-kube-api-access-nwmn4" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845284 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d578e40-60d8-4ebd-b0b5-2f5c274c7903" volumeName="kubernetes.io/configmap/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-certificates" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845291 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7321845c-9a95-415b-b6db-2bf983d77fa5" volumeName="kubernetes.io/secret/7321845c-9a95-415b-b6db-2bf983d77fa5-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845297 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82b3b5e2-09f1-49c0-8d72-c85187918223" volumeName="kubernetes.io/projected/82b3b5e2-09f1-49c0-8d72-c85187918223-kube-api-access-64dcq" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845304 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542" volumeName="kubernetes.io/secret/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542-networking-console-plugin-cert" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845313 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="071262de-534b-4d01-bc20-784db9852f69" volumeName="kubernetes.io/secret/071262de-534b-4d01-bc20-784db9852f69-serving-cert" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845320 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="215a42f8-881d-4cc4-b670-ea9f4aefb80b" volumeName="kubernetes.io/secret/215a42f8-881d-4cc4-b670-ea9f4aefb80b-cert" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845330 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26b7328d-dacf-44c2-995b-306e3ef9cf31" volumeName="kubernetes.io/projected/26b7328d-dacf-44c2-995b-306e3ef9cf31-kube-api-access-4zplm" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845350 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d578e40-60d8-4ebd-b0b5-2f5c274c7903" volumeName="kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-kube-api-access-gh6h5" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845359 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="457675c6-c725-44e5-a857-97aa4c17e655" volumeName="kubernetes.io/projected/457675c6-c725-44e5-a857-97aa4c17e655-kube-api-access-9gzjx" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845366 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="472c7524-6a70-420c-aa56-7f3028a53d52" volumeName="kubernetes.io/configmap/472c7524-6a70-420c-aa56-7f3028a53d52-iptables-alerter-script" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845373 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca5741f4-603c-448e-9088-0543eef08544" volumeName="kubernetes.io/projected/ca5741f4-603c-448e-9088-0543eef08544-kube-api-access-bjlw7" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845380 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff5ad4d1-4fd9-4374-b11c-78596c9cafa8" volumeName="kubernetes.io/projected/ff5ad4d1-4fd9-4374-b11c-78596c9cafa8-kube-api-access-cgmgr" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845387 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1dccc35b-6206-47f3-b901-bafc5f17b8ad" volumeName="kubernetes.io/configmap/1dccc35b-6206-47f3-b901-bafc5f17b8ad-service-ca-bundle" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845394 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="457675c6-c725-44e5-a857-97aa4c17e655" volumeName="kubernetes.io/configmap/457675c6-c725-44e5-a857-97aa4c17e655-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845401 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="566b2dae-dd3e-4c63-b4d2-e30667a5fd07" volumeName="kubernetes.io/projected/566b2dae-dd3e-4c63-b4d2-e30667a5fd07-kube-api-access-cqn4t" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845408 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7321845c-9a95-415b-b6db-2bf983d77fa5" volumeName="kubernetes.io/configmap/7321845c-9a95-415b-b6db-2bf983d77fa5-ovnkube-script-lib" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845416 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82b3b5e2-09f1-49c0-8d72-c85187918223" volumeName="kubernetes.io/secret/82b3b5e2-09f1-49c0-8d72-c85187918223-metrics-certs" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845423 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecdc26b5-be99-4928-8fff-203cc2c19ad6" volumeName="kubernetes.io/projected/ecdc26b5-be99-4928-8fff-203cc2c19ad6-kube-api-access-jjvmh" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845429 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc" volumeName="kubernetes.io/projected/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-kube-api-access-t6cmc" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845437 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7321845c-9a95-415b-b6db-2bf983d77fa5" volumeName="kubernetes.io/configmap/7321845c-9a95-415b-b6db-2bf983d77fa5-ovnkube-config" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845445 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7321845c-9a95-415b-b6db-2bf983d77fa5" volumeName="kubernetes.io/projected/7321845c-9a95-415b-b6db-2bf983d77fa5-kube-api-access-2t56h" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845451 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8d56c63-207c-42b3-8946-c993d0fde66b" volumeName="kubernetes.io/projected/a8d56c63-207c-42b3-8946-c993d0fde66b-kube-api-access-tnf2q" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845459 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae96a9c1-10a6-4050-bca0-6a2e7859d3c3" volumeName="kubernetes.io/secret/ae96a9c1-10a6-4050-bca0-6a2e7859d3c3-serving-cert" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845466 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc560bbe-b31a-48e8-8416-dda72f49dae1" volumeName="kubernetes.io/configmap/fc560bbe-b31a-48e8-8416-dda72f49dae1-serviceca" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845473 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1dccc35b-6206-47f3-b901-bafc5f17b8ad" volumeName="kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-stats-auth" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845479 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c" volumeName="kubernetes.io/projected/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-kube-api-access-zvp6g" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845508 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72ef59a1-3734-47b7-9074-f683ca94dd32" volumeName="kubernetes.io/configmap/72ef59a1-3734-47b7-9074-f683ca94dd32-config-volume" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845515 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d578e40-60d8-4ebd-b0b5-2f5c274c7903" volumeName="kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-bound-sa-token" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845525 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1dccc35b-6206-47f3-b901-bafc5f17b8ad" volumeName="kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-default-certificate" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845533 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26b7328d-dacf-44c2-995b-306e3ef9cf31" volumeName="kubernetes.io/configmap/26b7328d-dacf-44c2-995b-306e3ef9cf31-multus-daemon-config" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845541 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d578e40-60d8-4ebd-b0b5-2f5c274c7903" volumeName="kubernetes.io/configmap/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-trusted-ca" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845548 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc" volumeName="kubernetes.io/configmap/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-samples-operator-config" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845556 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ea8d3ef-20e9-4ecb-9e50-2df15d7da726" volumeName="kubernetes.io/secret/0ea8d3ef-20e9-4ecb-9e50-2df15d7da726-agent-certs" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845564 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5" volumeName="kubernetes.io/secret/a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5-serving-cert" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845577 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c" volumeName="kubernetes.io/empty-dir/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-tmp" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845584 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc560bbe-b31a-48e8-8416-dda72f49dae1" volumeName="kubernetes.io/projected/fc560bbe-b31a-48e8-8416-dda72f49dae1-kube-api-access-5t2t2" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845592 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13a7b3bb-060c-4383-a864-0575834b221e" volumeName="kubernetes.io/projected/13a7b3bb-060c-4383-a864-0575834b221e-kube-api-access-ht6kh" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845608 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53ba063f-0db3-4f82-b280-40718c28b479" volumeName="kubernetes.io/configmap/53ba063f-0db3-4f82-b280-40718c28b479-telemetry-config" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845615 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee0440bc-1614-4209-8744-6d197a019b5c" volumeName="kubernetes.io/secret/ee0440bc-1614-4209-8744-6d197a019b5c-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845621 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb045f55-e48d-4cac-88cf-90f90c225cbb" volumeName="kubernetes.io/projected/fb045f55-e48d-4cac-88cf-90f90c225cbb-kube-api-access-bfpxg" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845631 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13a7b3bb-060c-4383-a864-0575834b221e" volumeName="kubernetes.io/empty-dir/13a7b3bb-060c-4383-a864-0575834b221e-etc-tuned" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845638 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae96a9c1-10a6-4050-bca0-6a2e7859d3c3" volumeName="kubernetes.io/configmap/ae96a9c1-10a6-4050-bca0-6a2e7859d3c3-config" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845645 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb204d6b-3758-4db4-bd68-0eacec87b805" volumeName="kubernetes.io/projected/cb204d6b-3758-4db4-bd68-0eacec87b805-kube-api-access-njqr9" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845652 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee0440bc-1614-4209-8744-6d197a019b5c" volumeName="kubernetes.io/configmap/ee0440bc-1614-4209-8744-6d197a019b5c-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845661 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5" volumeName="kubernetes.io/configmap/a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5-trusted-ca" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845668 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13a7b3bb-060c-4383-a864-0575834b221e" volumeName="kubernetes.io/empty-dir/13a7b3bb-060c-4383-a864-0575834b221e-tmp" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845676 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1dccc35b-6206-47f3-b901-bafc5f17b8ad" volumeName="kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-metrics-certs" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845682 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d578e40-60d8-4ebd-b0b5-2f5c274c7903" volumeName="kubernetes.io/empty-dir/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-ca-trust-extracted" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845689 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="457675c6-c725-44e5-a857-97aa4c17e655" volumeName="kubernetes.io/configmap/457675c6-c725-44e5-a857-97aa4c17e655-cni-binary-copy" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845696 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="472c7524-6a70-420c-aa56-7f3028a53d52" volumeName="kubernetes.io/projected/472c7524-6a70-420c-aa56-7f3028a53d52-kube-api-access-l9bfd" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845703 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb204d6b-3758-4db4-bd68-0eacec87b805" volumeName="kubernetes.io/empty-dir/cb204d6b-3758-4db4-bd68-0eacec87b805-tmp-dir" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845709 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee0440bc-1614-4209-8744-6d197a019b5c" volumeName="kubernetes.io/empty-dir/ee0440bc-1614-4209-8744-6d197a019b5c-data-volume" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845718 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06f6f89f-b8f9-45ec-b50a-f3407bfe7ef8" volumeName="kubernetes.io/projected/06f6f89f-b8f9-45ec-b50a-f3407bfe7ef8-kube-api-access-b8t9t" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845733 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="071262de-534b-4d01-bc20-784db9852f69" volumeName="kubernetes.io/configmap/071262de-534b-4d01-bc20-784db9852f69-config" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845740 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26b7328d-dacf-44c2-995b-306e3ef9cf31" volumeName="kubernetes.io/configmap/26b7328d-dacf-44c2-995b-306e3ef9cf31-cni-binary-copy" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845747 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d578e40-60d8-4ebd-b0b5-2f5c274c7903" volumeName="kubernetes.io/secret/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-image-registry-private-configuration" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845755 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="457675c6-c725-44e5-a857-97aa4c17e655" volumeName="kubernetes.io/configmap/457675c6-c725-44e5-a857-97aa4c17e655-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845762 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c" volumeName="kubernetes.io/configmap/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-service-ca-bundle" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845770 5727 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c" volumeName="kubernetes.io/configmap/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-trusted-ca-bundle" seLinuxMountContext="" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845777 5727 reconstruct.go:97] "Volume reconstruction finished" Nov 02 22:48:33.846271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.845781 5727 reconciler.go:29] "Reconciler: start to sync state" Nov 02 22:48:33.868389 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.868355 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 22:48:33.869794 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.869771 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 22:48:33.873075 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.873039 5727 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 22:48:33.874457 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.874433 5727 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 22:48:33.874545 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.874458 5727 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 22:48:33.874545 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.874483 5727 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 22:48:33.874545 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:48:33.874526 5727 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 22:48:33.884029 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.884004 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 22:48:33.884169 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.884046 5727 generic.go:356] "Generic (PLEG): container finished" podID="a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5" containerID="de773a7c0efbc6b4377ddff49679469091bfed1e165b3ad6bfdfa9bd6760db62" exitCode=255 Nov 02 22:48:33.890096 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.890068 5727 generic.go:356] "Generic (PLEG): container finished" podID="049a11ea66e964e484c1cdf0430a7a18" containerID="71fc147a02e78f80153778c3094bee5085c45b4464333ee554fc6d780b478554" exitCode=0 Nov 02 22:48:33.919999 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.919947 5727 generic.go:356] "Generic (PLEG): container finished" podID="457675c6-c725-44e5-a857-97aa4c17e655" containerID="0d91fe2ba2bbe3d4b3f55c4ee701f5fb27f51bd238da250d54d46d81d4505723" exitCode=0 Nov 02 22:48:33.919999 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.920001 5727 generic.go:356] "Generic (PLEG): container finished" podID="457675c6-c725-44e5-a857-97aa4c17e655" containerID="2a408bee88ce403cbf74d4ef36e72f27962ed45de0c5b14df68f49cd1cc34728" exitCode=0 Nov 02 22:48:33.920303 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.920019 5727 generic.go:356] "Generic (PLEG): container finished" podID="457675c6-c725-44e5-a857-97aa4c17e655" containerID="249507aa03b4c41f387caa1326fcb1cd1b71ccef44b5321f9158deab71887385" exitCode=0 Nov 02 22:48:33.920303 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.920035 5727 generic.go:356] "Generic (PLEG): container finished" podID="457675c6-c725-44e5-a857-97aa4c17e655" containerID="eeb49842f57448e9799fdf4c141cba32728d039ca116aee8fe132ea734e04257" exitCode=0 Nov 02 22:48:33.920303 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.920048 5727 generic.go:356] "Generic (PLEG): container finished" podID="457675c6-c725-44e5-a857-97aa4c17e655" containerID="f550b5072b6e927ff4a84ccd13edcbc9ff98fa0aa1e9d415608123201a95bc3c" exitCode=0 Nov 02 22:48:33.920303 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.920058 5727 generic.go:356] "Generic (PLEG): container finished" podID="457675c6-c725-44e5-a857-97aa4c17e655" containerID="6cbea0ce2e4346e5fadef946dbea33fc4abb4b94e29de0dfc0411dc3199a0a65" exitCode=0 Nov 02 22:48:33.942087 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:48:33.942061 5727 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-176.us-west-1.compute.internal\" not found" Nov 02 22:48:33.955509 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.955487 5727 manager.go:324] Recovery completed Nov 02 22:48:33.974933 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:48:33.974869 5727 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 02 22:48:33.977419 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.977404 5727 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:48:33.979511 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.979492 5727 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-176.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:48:33.979617 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.979528 5727 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-176.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:48:33.979617 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.979542 5727 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-176.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:48:33.982979 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.982963 5727 cpu_manager.go:226] "Starting" policy="none" Nov 02 22:48:33.982979 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.982978 5727 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 22:48:33.983089 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.982994 5727 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 22:48:33.983089 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.982968 5727 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:48:33.983173 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.983117 5727 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 22:48:33.983173 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.983142 5727 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 22:48:33.983173 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.983158 5727 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 22:48:33.983173 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.983163 5727 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 22:48:33.983173 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.983169 5727 policy_none.go:50] "Start" Nov 02 22:48:33.983382 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.983177 5727 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 22:48:33.983382 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.983186 5727 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 22:48:33.983382 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.983322 5727 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 22:48:33.983382 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.983331 5727 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 22:48:33.983382 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.983336 5727 policy_none.go:44] "Start" Nov 02 22:48:33.983914 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.983896 5727 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-176.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:48:33.983983 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.983931 5727 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-176.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:48:33.983983 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.983946 5727 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-176.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:48:33.988276 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.988251 5727 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 22:48:33.990907 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.990895 5727 manager.go:341] "Starting Device Plugin manager" Nov 02 22:48:33.990959 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:48:33.990919 5727 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 22:48:33.990959 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.990928 5727 server.go:87] "Starting device plugin registration server" Nov 02 22:48:33.991202 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.991191 5727 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 22:48:33.991245 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.991203 5727 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 22:48:33.991340 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.991320 5727 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 22:48:33.991494 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.991458 5727 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 22:48:33.991551 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.991503 5727 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 22:48:33.991551 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.991510 5727 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 22:48:33.992235 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.992146 5727 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T22:48:33.991491906Z","UUID":"d1020de0-19b4-4d5d-a294-0f105b7337c6","Handler":null,"Name":"","Endpoint":""} Nov 02 22:48:33.993560 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:48:33.993536 5727 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 22:48:33.993671 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:48:33.993590 5727 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-176.us-west-1.compute.internal\" not found" Nov 02 22:48:33.996861 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.996841 5727 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 22:48:33.996946 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:33.996873 5727 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 22:48:34.036256 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.036229 5727 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:48:34.036349 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.036280 5727 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:48:34.036425 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.036348 5727 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:48:34.036488 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.036472 5727 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 22:48:34.086046 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.086021 5727 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:48:34.092155 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.092137 5727 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:48:34.093107 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.093091 5727 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-176.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:48:34.093206 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.093147 5727 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-176.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:48:34.093206 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.093163 5727 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-176.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:48:34.093316 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.093218 5727 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:48:34.175731 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.175657 5727 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-176.us-west-1.compute.internal"] Nov 02 22:48:34.176383 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.176340 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal" event={"ID":"049a11ea66e964e484c1cdf0430a7a18","Type":"ContainerStarted","Data":"f15dc2bdc14df045bd1ac6f37ddb5a899d3f9c6658de31febddcfee628bfe087"} Nov 02 22:48:34.176465 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.176402 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal" event={"ID":"049a11ea66e964e484c1cdf0430a7a18","Type":"ContainerDied","Data":"71fc147a02e78f80153778c3094bee5085c45b4464333ee554fc6d780b478554"} Nov 02 22:48:34.176465 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.176421 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal" event={"ID":"049a11ea66e964e484c1cdf0430a7a18","Type":"ContainerStarted","Data":"2d0f15b28f79d9d966377869a930b9cd17054031a166bfbbd0473f25097a1d2a"} Nov 02 22:48:34.176465 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.176442 5727 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:48:34.176630 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.176497 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.us-west-1.compute.internal" event={"ID":"38d433b21a6573d8bddad58877b72617","Type":"ContainerStarted","Data":"d670b52d4d4290795d2163b7672306d251c56590fd22e98d773214ad90aee696"} Nov 02 22:48:34.176630 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.176514 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.us-west-1.compute.internal" event={"ID":"38d433b21a6573d8bddad58877b72617","Type":"ContainerStarted","Data":"149bfac297d1959995d6a77e8d0cb2e89a62a5efe665a352f8dd48c6a210de40"} Nov 02 22:48:34.176630 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.176567 5727 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:48:34.236826 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.236764 5727 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:48:34.236944 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:48:34.236840 5727 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-176.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:48:34.237231 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.237215 5727 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:48:34.237304 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:48:34.237246 5727 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:48:34.241120 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.241102 5727 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:48:34.241239 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.241191 5727 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:48:34.337030 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.337003 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/049a11ea66e964e484c1cdf0430a7a18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal\" (UID: \"049a11ea66e964e484c1cdf0430a7a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:48:34.337144 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.337031 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/049a11ea66e964e484c1cdf0430a7a18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal\" (UID: \"049a11ea66e964e484c1cdf0430a7a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:48:34.337144 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.337049 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/38d433b21a6573d8bddad58877b72617-config\") pod \"kube-apiserver-proxy-ip-10-0-135-176.us-west-1.compute.internal\" (UID: \"38d433b21a6573d8bddad58877b72617\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:48:34.437653 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.437628 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/049a11ea66e964e484c1cdf0430a7a18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal\" (UID: \"049a11ea66e964e484c1cdf0430a7a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:48:34.437783 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.437657 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/049a11ea66e964e484c1cdf0430a7a18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal\" (UID: \"049a11ea66e964e484c1cdf0430a7a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:48:34.437783 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.437683 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/38d433b21a6573d8bddad58877b72617-config\") pod \"kube-apiserver-proxy-ip-10-0-135-176.us-west-1.compute.internal\" (UID: \"38d433b21a6573d8bddad58877b72617\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:48:34.437783 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.437735 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/049a11ea66e964e484c1cdf0430a7a18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal\" (UID: \"049a11ea66e964e484c1cdf0430a7a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:48:34.437783 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.437751 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/049a11ea66e964e484c1cdf0430a7a18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal\" (UID: \"049a11ea66e964e484c1cdf0430a7a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:48:34.437928 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.437804 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/38d433b21a6573d8bddad58877b72617-config\") pod \"kube-apiserver-proxy-ip-10-0-135-176.us-west-1.compute.internal\" (UID: \"38d433b21a6573d8bddad58877b72617\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:48:34.825437 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.825406 5727 apiserver.go:52] "Watching apiserver" Nov 02 22:48:34.929096 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.929065 5727 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 22:48:34.929846 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.929823 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-wbxxp","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5ljqt","openshift-image-registry/node-ca-mp88w","openshift-network-console/networking-console-plugin-86c478bb88-nj7zh","openshift-network-diagnostics/network-check-target-2qpkq","openshift-ingress-canary/ingress-canary-6jd4b","openshift-service-ca-operator/service-ca-operator-544d749c8c-xglvd","openshift-multus/multus-mcvpd","kube-system/konnectivity-agent-r5lnw","kube-system/kas-connection-checker-685895c7c9-5wnxz","openshift-insights/insights-runtime-extractor-qlfjm","kube-system/kas-connection-checker-685895c7c9-276wh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd","openshift-image-registry/image-registry-bc8cbc8c-hghqr","openshift-network-operator/iptables-alerter-rw8v4","openshift-console-operator/console-operator-7cd56b8dbc-fj4n7","openshift-network-diagnostics/network-check-source-6f78d59f99-nbswd","openshift-dns/node-resolver-lhbwd","openshift-kube-storage-version-migrator/migrator-66df5d496b-ttdqt","openshift-multus/network-metrics-daemon-465bf","openshift-monitoring/cluster-monitoring-operator-884576c57-qcz4m","openshift-cluster-node-tuning-operator/tuned-2kt76","openshift-ovn-kubernetes/ovnkube-node-pl8m7","openshift-dns/dns-default-c8kg9","kube-system/kube-apiserver-proxy-ip-10-0-135-176.us-west-1.compute.internal","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rf5x6","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt","kube-system/kas-connection-checker-685895c7c9-25msk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal","openshift-ingress/router-default-566bb4455-kq2j4","openshift-insights/insights-operator-7c4986d449-464jz","openshift-multus/multus-additional-cni-plugins-t9rnl"] Nov 02 22:48:34.946514 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.946492 5727 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:48:34.946643 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.946585 5727 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:48:34.982814 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.982794 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 02 22:48:34.982896 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.982810 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 02 22:48:34.983066 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.983052 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 02 22:48:34.983871 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.983850 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 02 22:48:34.983941 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.983891 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 22:48:34.983992 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.983938 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:48:34.983992 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.983947 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 22:48:34.984091 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.984074 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 02 22:48:34.984176 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.984147 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 02 22:48:34.984176 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.984155 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 02 22:48:34.984398 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.984372 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 02 22:48:34.984461 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.984374 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 02 22:48:34.984509 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.984486 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 02 22:48:34.984594 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.984578 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 02 22:48:34.984654 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.984585 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:48:34.984654 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.984588 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:48:34.984654 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.984635 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 02 22:48:34.984984 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.984953 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 02 22:48:34.984984 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.984984 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 02 22:48:34.985117 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.985075 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 02 22:48:34.985117 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.985090 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:48:34.985233 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.985217 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 02 22:48:34.985284 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.985218 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 02 22:48:34.985472 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.985457 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 22:48:34.985608 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.985596 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 02 22:48:34.985886 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.985873 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 02 22:48:34.985920 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.985906 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 02 22:48:34.986195 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:34.986181 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 22:48:35.036303 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.036263 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 22:48:35.036303 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.036265 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 22:48:35.036303 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.036290 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 22:48:35.036303 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.036307 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 22:48:35.036567 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.036390 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 02 22:48:35.036567 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.036403 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 02 22:48:35.036567 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.036417 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 22:48:35.036567 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.036437 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 22:48:35.036567 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.036480 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 02 22:48:35.036567 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.036519 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 22:48:35.036567 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.036542 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 22:48:35.036852 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.036617 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 02 22:48:35.036852 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.036639 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:48:35.036852 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.036665 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9jqgm\"" Nov 02 22:48:35.036852 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.036708 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f4f84\"" Nov 02 22:48:35.036852 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.036747 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-77tzt\"" Nov 02 22:48:35.036852 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.036764 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 22:48:35.036852 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.036782 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 22:48:35.036852 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.036792 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zfnv8\"" Nov 02 22:48:35.036852 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.036819 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qf97t\"" Nov 02 22:48:35.036852 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.036847 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-cv7fc\"" Nov 02 22:48:35.037300 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.036865 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-pl59x\"" Nov 02 22:48:35.037300 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.036881 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 22:48:35.037300 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.036927 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p2dkz\"" Nov 02 22:48:35.037300 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.036989 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-rncv6\"" Nov 02 22:48:35.037300 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037058 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 22:48:35.037300 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037103 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 22:48:35.037300 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037149 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 22:48:35.037300 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037184 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 22:48:35.037300 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037188 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 22:48:35.037300 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037209 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 22:48:35.037300 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037221 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 22:48:35.037300 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037213 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 22:48:35.037300 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037249 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 22:48:35.037300 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037250 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 22:48:35.037300 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037262 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 22:48:35.037300 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037260 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5z48j\"" Nov 02 22:48:35.037300 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037281 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 22:48:35.037300 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037287 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 22:48:35.037300 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037302 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 22:48:35.037300 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037315 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 22:48:35.038228 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037353 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 22:48:35.038228 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037365 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 22:48:35.038228 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037393 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 22:48:35.038228 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037410 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 22:48:35.038228 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037422 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 22:48:35.038228 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037430 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r8wwr\"" Nov 02 22:48:35.038228 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037438 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 22:48:35.038228 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037458 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 22:48:35.038228 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037464 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 22:48:35.038228 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037477 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wk6k8\"" Nov 02 22:48:35.038228 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037481 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 22:48:35.038228 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037508 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bxrdd\"" Nov 02 22:48:35.038228 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037534 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 22:48:35.038228 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037602 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:48:35.038228 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037616 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hnmtx\"" Nov 02 22:48:35.038228 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037667 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-d58gp\"" Nov 02 22:48:35.038228 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037703 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 22:48:35.038228 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037715 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mb4zk\"" Nov 02 22:48:35.038228 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037761 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 22:48:35.038228 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037796 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:48:35.038228 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037835 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 22:48:35.038228 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037865 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 22:48:35.038228 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037950 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 22:48:35.038228 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037984 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 22:48:35.038228 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.037986 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 22:48:35.038228 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.038051 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9g8qt\"" Nov 02 22:48:35.038228 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.038088 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 02 22:48:35.038228 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.038108 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 22:48:35.038228 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.038168 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 22:48:35.038228 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.038239 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-cz5vj\"" Nov 02 22:48:35.039060 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.038327 5727 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:48:35.039060 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:48:35.038373 5727 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:48:35.039060 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.038415 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 22:48:35.039060 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.038890 5727 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:48:35.039060 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:48:35.038923 5727 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-176.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.us-west-1.compute.internal" Nov 02 22:48:35.040038 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040021 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-systemd\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.040103 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040057 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbvvt\" (UniqueName: \"kubernetes.io/projected/53ba063f-0db3-4f82-b280-40718c28b479-kube-api-access-jbvvt\") pod \"cluster-monitoring-operator-884576c57-qcz4m\" (UID: \"53ba063f-0db3-4f82-b280-40718c28b479\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qcz4m" Nov 02 22:48:35.040103 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040087 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-kubelet\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.040247 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040142 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-sysctl-conf\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.040247 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040194 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7321845c-9a95-415b-b6db-2bf983d77fa5-ovnkube-config\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.040247 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040235 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-tmp\") pod \"insights-operator-7c4986d449-464jz\" (UID: \"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c\") " pod="openshift-insights/insights-operator-7c4986d449-464jz" Nov 02 22:48:35.040386 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040253 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfpxg\" (UniqueName: \"kubernetes.io/projected/fb045f55-e48d-4cac-88cf-90f90c225cbb-kube-api-access-bfpxg\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:48:35.040386 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040324 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-tmp\") pod \"insights-operator-7c4986d449-464jz\" (UID: \"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c\") " pod="openshift-insights/insights-operator-7c4986d449-464jz" Nov 02 22:48:35.040386 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040348 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-cni-bin\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.040537 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040390 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9bfd\" (UniqueName: \"kubernetes.io/projected/472c7524-6a70-420c-aa56-7f3028a53d52-kube-api-access-l9bfd\") pod \"iptables-alerter-rw8v4\" (UID: \"472c7524-6a70-420c-aa56-7f3028a53d52\") " pod="openshift-network-operator/iptables-alerter-rw8v4" Nov 02 22:48:35.040537 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040431 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjlw7\" (UniqueName: \"kubernetes.io/projected/ca5741f4-603c-448e-9088-0543eef08544-kube-api-access-bjlw7\") pod \"volume-data-source-validator-58c9f5f46d-5ljqt\" (UID: \"ca5741f4-603c-448e-9088-0543eef08544\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5ljqt" Nov 02 22:48:35.040537 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040455 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-stats-auth\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:48:35.040537 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040481 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7321845c-9a95-415b-b6db-2bf983d77fa5-ovn-node-metrics-cert\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.040537 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040497 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-multus-conf-dir\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.040537 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040513 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26b7328d-dacf-44c2-995b-306e3ef9cf31-multus-daemon-config\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.040537 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040531 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ba063f-0db3-4f82-b280-40718c28b479-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qcz4m\" (UID: \"53ba063f-0db3-4f82-b280-40718c28b479\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qcz4m" Nov 02 22:48:35.040864 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040553 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7321845c-9a95-415b-b6db-2bf983d77fa5-ovnkube-config\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.040864 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040556 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7321845c-9a95-415b-b6db-2bf983d77fa5-env-overrides\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.040864 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040594 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-service-ca-bundle\") pod \"insights-operator-7c4986d449-464jz\" (UID: \"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c\") " pod="openshift-insights/insights-operator-7c4986d449-464jz" Nov 02 22:48:35.040864 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040621 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-host\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.040864 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040644 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/457675c6-c725-44e5-a857-97aa4c17e655-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:48:35.040864 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040671 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/457675c6-c725-44e5-a857-97aa4c17e655-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:48:35.040864 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040708 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-sys-fs\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:48:35.040864 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040739 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5-config\") pod \"console-operator-7cd56b8dbc-fj4n7\" (UID: \"a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" Nov 02 22:48:35.040864 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040757 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26b7328d-dacf-44c2-995b-306e3ef9cf31-multus-daemon-config\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.040864 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040766 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7321845c-9a95-415b-b6db-2bf983d77fa5-ovn-node-metrics-cert\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.040864 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040767 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb204d6b-3758-4db4-bd68-0eacec87b805-hosts-file\") pod \"node-resolver-lhbwd\" (UID: \"cb204d6b-3758-4db4-bd68-0eacec87b805\") " pod="openshift-dns/node-resolver-lhbwd" Nov 02 22:48:35.040864 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040791 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-stats-auth\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:48:35.040864 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040804 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ba063f-0db3-4f82-b280-40718c28b479-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qcz4m\" (UID: \"53ba063f-0db3-4f82-b280-40718c28b479\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qcz4m" Nov 02 22:48:35.040864 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040813 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-sys\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.040864 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040845 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae96a9c1-10a6-4050-bca0-6a2e7859d3c3-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt\" (UID: \"ae96a9c1-10a6-4050-bca0-6a2e7859d3c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt" Nov 02 22:48:35.040864 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040855 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/457675c6-c725-44e5-a857-97aa4c17e655-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:48:35.041530 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040874 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-hostroot\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.041530 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040905 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqn4t\" (UniqueName: \"kubernetes.io/projected/566b2dae-dd3e-4c63-b4d2-e30667a5fd07-kube-api-access-cqn4t\") pod \"network-check-target-2qpkq\" (UID: \"566b2dae-dd3e-4c63-b4d2-e30667a5fd07\") " pod="openshift-network-diagnostics/network-check-target-2qpkq" Nov 02 22:48:35.041530 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040953 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5-config\") pod \"console-operator-7cd56b8dbc-fj4n7\" (UID: \"a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" Nov 02 22:48:35.041530 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040929 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-run-netns\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.041530 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.040998 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/417d26e1-2a9d-41e3-8940-69626662a2f3-original-pull-secret\") pod \"global-pull-secret-syncer-wbxxp\" (UID: \"417d26e1-2a9d-41e3-8940-69626662a2f3\") " pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:48:35.041530 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041027 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-snapshots\") pod \"insights-operator-7c4986d449-464jz\" (UID: \"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c\") " pod="openshift-insights/insights-operator-7c4986d449-464jz" Nov 02 22:48:35.041530 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041029 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7321845c-9a95-415b-b6db-2bf983d77fa5-env-overrides\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.041530 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041085 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb204d6b-3758-4db4-bd68-0eacec87b805-tmp-dir\") pod \"node-resolver-lhbwd\" (UID: \"cb204d6b-3758-4db4-bd68-0eacec87b805\") " pod="openshift-dns/node-resolver-lhbwd" Nov 02 22:48:35.041530 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041165 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zplm\" (UniqueName: \"kubernetes.io/projected/26b7328d-dacf-44c2-995b-306e3ef9cf31-kube-api-access-4zplm\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.041530 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041177 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb204d6b-3758-4db4-bd68-0eacec87b805-tmp-dir\") pod \"node-resolver-lhbwd\" (UID: \"cb204d6b-3758-4db4-bd68-0eacec87b805\") " pod="openshift-dns/node-resolver-lhbwd" Nov 02 22:48:35.041530 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041168 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-snapshots\") pod \"insights-operator-7c4986d449-464jz\" (UID: \"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c\") " pod="openshift-insights/insights-operator-7c4986d449-464jz" Nov 02 22:48:35.041530 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041192 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae96a9c1-10a6-4050-bca0-6a2e7859d3c3-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt\" (UID: \"ae96a9c1-10a6-4050-bca0-6a2e7859d3c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt" Nov 02 22:48:35.041530 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041210 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/457675c6-c725-44e5-a857-97aa4c17e655-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:48:35.041530 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041222 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-service-ca-bundle\") pod \"insights-operator-7c4986d449-464jz\" (UID: \"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c\") " pod="openshift-insights/insights-operator-7c4986d449-464jz" Nov 02 22:48:35.041530 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041230 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-sysconfig\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.041530 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041264 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-registration-dir\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:48:35.041530 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041287 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rn6f\" (UniqueName: \"kubernetes.io/projected/071262de-534b-4d01-bc20-784db9852f69-kube-api-access-7rn6f\") pod \"service-ca-operator-544d749c8c-xglvd\" (UID: \"071262de-534b-4d01-bc20-784db9852f69\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xglvd" Nov 02 22:48:35.041530 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041303 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-kubernetes\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.041530 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041342 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0ea8d3ef-20e9-4ecb-9e50-2df15d7da726-konnectivity-ca\") pod \"konnectivity-agent-r5lnw\" (UID: \"0ea8d3ef-20e9-4ecb-9e50-2df15d7da726\") " pod="kube-system/konnectivity-agent-r5lnw" Nov 02 22:48:35.041530 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041357 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/417d26e1-2a9d-41e3-8940-69626662a2f3-original-pull-secret\") pod \"global-pull-secret-syncer-wbxxp\" (UID: \"417d26e1-2a9d-41e3-8940-69626662a2f3\") " pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:48:35.041530 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041387 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/457675c6-c725-44e5-a857-97aa4c17e655-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:48:35.041530 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041433 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/457675c6-c725-44e5-a857-97aa4c17e655-system-cni-dir\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:48:35.041530 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041481 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/53ba063f-0db3-4f82-b280-40718c28b479-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-qcz4m\" (UID: \"53ba063f-0db3-4f82-b280-40718c28b479\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qcz4m" Nov 02 22:48:35.041530 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041504 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-systemd-units\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.041530 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041539 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfz5j\" (UniqueName: \"kubernetes.io/projected/a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5-kube-api-access-bfz5j\") pod \"console-operator-7cd56b8dbc-fj4n7\" (UID: \"a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" Nov 02 22:48:35.042507 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041565 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0ea8d3ef-20e9-4ecb-9e50-2df15d7da726-konnectivity-ca\") pod \"konnectivity-agent-r5lnw\" (UID: \"0ea8d3ef-20e9-4ecb-9e50-2df15d7da726\") " pod="kube-system/konnectivity-agent-r5lnw" Nov 02 22:48:35.042507 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041570 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsmhx\" (UniqueName: \"kubernetes.io/projected/12ce41df-43ef-4b00-be8e-5d3fad591eea-kube-api-access-gsmhx\") pod \"kas-connection-checker-685895c7c9-276wh\" (UID: \"12ce41df-43ef-4b00-be8e-5d3fad591eea\") " pod="kube-system/kas-connection-checker-685895c7c9-276wh" Nov 02 22:48:35.042507 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041611 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26b7328d-dacf-44c2-995b-306e3ef9cf31-cni-binary-copy\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.042507 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041639 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-var-lib-kubelet\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.042507 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041665 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-device-dir\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:48:35.042507 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041691 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgmgr\" (UniqueName: \"kubernetes.io/projected/ff5ad4d1-4fd9-4374-b11c-78596c9cafa8-kube-api-access-cgmgr\") pod \"network-check-source-6f78d59f99-nbswd\" (UID: \"ff5ad4d1-4fd9-4374-b11c-78596c9cafa8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nbswd" Nov 02 22:48:35.042507 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041697 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/53ba063f-0db3-4f82-b280-40718c28b479-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-qcz4m\" (UID: \"53ba063f-0db3-4f82-b280-40718c28b479\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qcz4m" Nov 02 22:48:35.042507 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041732 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qw9l\" (UniqueName: \"kubernetes.io/projected/ae96a9c1-10a6-4050-bca0-6a2e7859d3c3-kube-api-access-9qw9l\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt\" (UID: \"ae96a9c1-10a6-4050-bca0-6a2e7859d3c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt" Nov 02 22:48:35.042507 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041766 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/417d26e1-2a9d-41e3-8940-69626662a2f3-dbus\") pod \"global-pull-secret-syncer-wbxxp\" (UID: \"417d26e1-2a9d-41e3-8940-69626662a2f3\") " pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:48:35.042507 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041814 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26b7328d-dacf-44c2-995b-306e3ef9cf31-cni-binary-copy\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.042507 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041833 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-var-lib-kubelet\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.042507 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041870 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/13a7b3bb-060c-4383-a864-0575834b221e-etc-tuned\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.042507 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041900 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13a7b3bb-060c-4383-a864-0575834b221e-tmp\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.042507 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041923 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:48:35.042507 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041947 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-run-ovn\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.042507 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041980 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13a7b3bb-060c-4383-a864-0575834b221e-tmp\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.042507 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041986 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7321845c-9a95-415b-b6db-2bf983d77fa5-ovnkube-script-lib\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.042507 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.041985 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/13a7b3bb-060c-4383-a864-0575834b221e-etc-tuned\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.042507 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.042023 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-lib-modules\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.042507 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.042108 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht6kh\" (UniqueName: \"kubernetes.io/projected/13a7b3bb-060c-4383-a864-0575834b221e-kube-api-access-ht6kh\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.042507 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.042142 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5bg9\" (UniqueName: \"kubernetes.io/projected/1dccc35b-6206-47f3-b901-bafc5f17b8ad-kube-api-access-s5bg9\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:48:35.042507 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.042158 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-slash\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.042507 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.042180 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7321845c-9a95-415b-b6db-2bf983d77fa5-ovnkube-script-lib\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.042507 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.042256 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-var-lib-openvswitch\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.042507 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.042281 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fc560bbe-b31a-48e8-8416-dda72f49dae1-serviceca\") pod \"node-ca-mp88w\" (UID: \"fc560bbe-b31a-48e8-8416-dda72f49dae1\") " pod="openshift-image-registry/node-ca-mp88w" Nov 02 22:48:35.042507 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.042378 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-etc-openvswitch\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.042507 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.042431 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-cni-netd\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.042507 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.042488 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/472c7524-6a70-420c-aa56-7f3028a53d52-iptables-alerter-script\") pod \"iptables-alerter-rw8v4\" (UID: \"472c7524-6a70-420c-aa56-7f3028a53d52\") " pod="openshift-network-operator/iptables-alerter-rw8v4" Nov 02 22:48:35.042507 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.042512 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-system-cni-dir\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.042529 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-etc-selinux\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.042573 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/071262de-534b-4d01-bc20-784db9852f69-config\") pod \"service-ca-operator-544d749c8c-xglvd\" (UID: \"071262de-534b-4d01-bc20-784db9852f69\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xglvd" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.042633 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64dcq\" (UniqueName: \"kubernetes.io/projected/82b3b5e2-09f1-49c0-8d72-c85187918223-kube-api-access-64dcq\") pod \"network-metrics-daemon-465bf\" (UID: \"82b3b5e2-09f1-49c0-8d72-c85187918223\") " pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.042686 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/472c7524-6a70-420c-aa56-7f3028a53d52-iptables-alerter-script\") pod \"iptables-alerter-rw8v4\" (UID: \"472c7524-6a70-420c-aa56-7f3028a53d52\") " pod="openshift-network-operator/iptables-alerter-rw8v4" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.042702 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjvmh\" (UniqueName: \"kubernetes.io/projected/ecdc26b5-be99-4928-8fff-203cc2c19ad6-kube-api-access-jjvmh\") pod \"kas-connection-checker-685895c7c9-25msk\" (UID: \"ecdc26b5-be99-4928-8fff-203cc2c19ad6\") " pod="kube-system/kas-connection-checker-685895c7c9-25msk" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.042761 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae96a9c1-10a6-4050-bca0-6a2e7859d3c3-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt\" (UID: \"ae96a9c1-10a6-4050-bca0-6a2e7859d3c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.042793 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82b3b5e2-09f1-49c0-8d72-c85187918223-metrics-certs\") pod \"network-metrics-daemon-465bf\" (UID: \"82b3b5e2-09f1-49c0-8d72-c85187918223\") " pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.042810 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/071262de-534b-4d01-bc20-784db9852f69-config\") pod \"service-ca-operator-544d749c8c-xglvd\" (UID: \"071262de-534b-4d01-bc20-784db9852f69\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xglvd" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.042842 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-run-netns\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.042888 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fc560bbe-b31a-48e8-8416-dda72f49dae1-serviceca\") pod \"node-ca-mp88w\" (UID: \"fc560bbe-b31a-48e8-8416-dda72f49dae1\") " pod="openshift-image-registry/node-ca-mp88w" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.042975 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae96a9c1-10a6-4050-bca0-6a2e7859d3c3-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt\" (UID: \"ae96a9c1-10a6-4050-bca0-6a2e7859d3c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.042999 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82b3b5e2-09f1-49c0-8d72-c85187918223-metrics-certs\") pod \"network-metrics-daemon-465bf\" (UID: \"82b3b5e2-09f1-49c0-8d72-c85187918223\") " pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.043008 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-run-ovn-kubernetes\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.043035 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njqr9\" (UniqueName: \"kubernetes.io/projected/cb204d6b-3758-4db4-bd68-0eacec87b805-kube-api-access-njqr9\") pod \"node-resolver-lhbwd\" (UID: \"cb204d6b-3758-4db4-bd68-0eacec87b805\") " pod="openshift-dns/node-resolver-lhbwd" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.043057 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-default-certificate\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.043143 5727 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.043155 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-run\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.043182 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-cnibin\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.043208 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-multus-socket-dir-parent\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.043240 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-default-certificate\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.043262 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-run-k8s-cni-cncf-io\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.043287 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-socket-dir\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.043312 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-node-log\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.043336 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.043366 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-464jz\" (UID: \"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c\") " pod="openshift-insights/insights-operator-7c4986d449-464jz" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.043408 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6cmc\" (UniqueName: \"kubernetes.io/projected/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-kube-api-access-t6cmc\") pod \"cluster-samples-operator-6ddb6c8b94-rf5x6\" (UID: \"2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rf5x6" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.043439 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-run-openvswitch\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.043471 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t56h\" (UniqueName: \"kubernetes.io/projected/7321845c-9a95-415b-b6db-2bf983d77fa5-kube-api-access-2t56h\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.043497 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/472c7524-6a70-420c-aa56-7f3028a53d52-host-slash\") pod \"iptables-alerter-rw8v4\" (UID: \"472c7524-6a70-420c-aa56-7f3028a53d52\") " pod="openshift-network-operator/iptables-alerter-rw8v4" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.043523 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5-trusted-ca\") pod \"console-operator-7cd56b8dbc-fj4n7\" (UID: \"a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.043549 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nj7zh\" (UID: \"d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nj7zh" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.043592 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-rf5x6\" (UID: \"2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rf5x6" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.043628 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-multus-cni-dir\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.043687 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-etc-kubernetes\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.043744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.043718 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/071262de-534b-4d01-bc20-784db9852f69-serving-cert\") pod \"service-ca-operator-544d749c8c-xglvd\" (UID: \"071262de-534b-4d01-bc20-784db9852f69\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xglvd" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.043769 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nj7zh\" (UID: \"d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nj7zh" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.043775 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnf2q\" (UniqueName: \"kubernetes.io/projected/a8d56c63-207c-42b3-8946-c993d0fde66b-kube-api-access-tnf2q\") pod \"kas-connection-checker-685895c7c9-5wnxz\" (UID: \"a8d56c63-207c-42b3-8946-c993d0fde66b\") " pod="kube-system/kas-connection-checker-685895c7c9-5wnxz" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.043773 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-rf5x6\" (UID: \"2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rf5x6" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.043858 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/417d26e1-2a9d-41e3-8940-69626662a2f3-kubelet-config\") pod \"global-pull-secret-syncer-wbxxp\" (UID: \"417d26e1-2a9d-41e3-8940-69626662a2f3\") " pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.043967 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-serving-cert\") pod \"insights-operator-7c4986d449-464jz\" (UID: \"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c\") " pod="openshift-insights/insights-operator-7c4986d449-464jz" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.044035 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542-nginx-conf\") pod \"networking-console-plugin-86c478bb88-nj7zh\" (UID: \"d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nj7zh" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.044054 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/071262de-534b-4d01-bc20-784db9852f69-serving-cert\") pod \"service-ca-operator-544d749c8c-xglvd\" (UID: \"071262de-534b-4d01-bc20-784db9852f69\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xglvd" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.044093 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rf5x6\" (UID: \"2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rf5x6" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.044105 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-464jz\" (UID: \"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c\") " pod="openshift-insights/insights-operator-7c4986d449-464jz" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.044117 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gzjx\" (UniqueName: \"kubernetes.io/projected/457675c6-c725-44e5-a857-97aa4c17e655-kube-api-access-9gzjx\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.044161 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvp6g\" (UniqueName: \"kubernetes.io/projected/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-kube-api-access-zvp6g\") pod \"insights-operator-7c4986d449-464jz\" (UID: \"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c\") " pod="openshift-insights/insights-operator-7c4986d449-464jz" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.044186 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-sysctl-d\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.044216 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-metrics-certs\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.044224 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542-nginx-conf\") pod \"networking-console-plugin-86c478bb88-nj7zh\" (UID: \"d9fd0cb2-d80d-4287-b9ca-d8b1d3d75542\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nj7zh" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.044239 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-var-lib-cni-bin\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.044267 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5-trusted-ca\") pod \"console-operator-7cd56b8dbc-fj4n7\" (UID: \"a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.044292 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t2t2\" (UniqueName: \"kubernetes.io/projected/fc560bbe-b31a-48e8-8416-dda72f49dae1-kube-api-access-5t2t2\") pod \"node-ca-mp88w\" (UID: \"fc560bbe-b31a-48e8-8416-dda72f49dae1\") " pod="openshift-image-registry/node-ca-mp88w" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.044332 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rf5x6\" (UID: \"2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rf5x6" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.044349 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/457675c6-c725-44e5-a857-97aa4c17e655-cni-binary-copy\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.044447 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dccc35b-6206-47f3-b901-bafc5f17b8ad-metrics-certs\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.044448 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-serving-cert\") pod \"insights-operator-7c4986d449-464jz\" (UID: \"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c\") " pod="openshift-insights/insights-operator-7c4986d449-464jz" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.044479 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5-serving-cert\") pod \"console-operator-7cd56b8dbc-fj4n7\" (UID: \"a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.044511 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-var-lib-cni-multus\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.044538 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0ea8d3ef-20e9-4ecb-9e50-2df15d7da726-agent-certs\") pod \"konnectivity-agent-r5lnw\" (UID: \"0ea8d3ef-20e9-4ecb-9e50-2df15d7da726\") " pod="kube-system/konnectivity-agent-r5lnw" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.044562 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-modprobe-d\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.044586 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-log-socket\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.044609 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/457675c6-c725-44e5-a857-97aa4c17e655-cnibin\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.044609 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/457675c6-c725-44e5-a857-97aa4c17e655-cni-binary-copy\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.044644 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc560bbe-b31a-48e8-8416-dda72f49dae1-host\") pod \"node-ca-mp88w\" (UID: \"fc560bbe-b31a-48e8-8416-dda72f49dae1\") " pod="openshift-image-registry/node-ca-mp88w" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.044735 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-run-systemd\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.044773 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dccc35b-6206-47f3-b901-bafc5f17b8ad-service-ca-bundle\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.044848 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-run-multus-certs\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.044846 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0ea8d3ef-20e9-4ecb-9e50-2df15d7da726-agent-certs\") pod \"konnectivity-agent-r5lnw\" (UID: \"0ea8d3ef-20e9-4ecb-9e50-2df15d7da726\") " pod="kube-system/konnectivity-agent-r5lnw" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.044866 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5-serving-cert\") pod \"console-operator-7cd56b8dbc-fj4n7\" (UID: \"a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" Nov 02 22:48:35.045015 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.044935 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dccc35b-6206-47f3-b901-bafc5f17b8ad-service-ca-bundle\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:48:35.098658 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.098607 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbvvt\" (UniqueName: \"kubernetes.io/projected/53ba063f-0db3-4f82-b280-40718c28b479-kube-api-access-jbvvt\") pod \"cluster-monitoring-operator-884576c57-qcz4m\" (UID: \"53ba063f-0db3-4f82-b280-40718c28b479\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qcz4m" Nov 02 22:48:35.098658 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.098633 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfpxg\" (UniqueName: \"kubernetes.io/projected/fb045f55-e48d-4cac-88cf-90f90c225cbb-kube-api-access-bfpxg\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:48:35.098658 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.098609 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9bfd\" (UniqueName: \"kubernetes.io/projected/472c7524-6a70-420c-aa56-7f3028a53d52-kube-api-access-l9bfd\") pod \"iptables-alerter-rw8v4\" (UID: \"472c7524-6a70-420c-aa56-7f3028a53d52\") " pod="openshift-network-operator/iptables-alerter-rw8v4" Nov 02 22:48:35.098850 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.098706 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjlw7\" (UniqueName: \"kubernetes.io/projected/ca5741f4-603c-448e-9088-0543eef08544-kube-api-access-bjlw7\") pod \"volume-data-source-validator-58c9f5f46d-5ljqt\" (UID: \"ca5741f4-603c-448e-9088-0543eef08544\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5ljqt" Nov 02 22:48:35.099089 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.099069 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zplm\" (UniqueName: \"kubernetes.io/projected/26b7328d-dacf-44c2-995b-306e3ef9cf31-kube-api-access-4zplm\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.099464 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.099445 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqn4t\" (UniqueName: \"kubernetes.io/projected/566b2dae-dd3e-4c63-b4d2-e30667a5fd07-kube-api-access-cqn4t\") pod \"network-check-target-2qpkq\" (UID: \"566b2dae-dd3e-4c63-b4d2-e30667a5fd07\") " pod="openshift-network-diagnostics/network-check-target-2qpkq" Nov 02 22:48:35.099836 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.099819 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsmhx\" (UniqueName: \"kubernetes.io/projected/12ce41df-43ef-4b00-be8e-5d3fad591eea-kube-api-access-gsmhx\") pod \"kas-connection-checker-685895c7c9-276wh\" (UID: \"12ce41df-43ef-4b00-be8e-5d3fad591eea\") " pod="kube-system/kas-connection-checker-685895c7c9-276wh" Nov 02 22:48:35.100581 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.100560 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5bg9\" (UniqueName: \"kubernetes.io/projected/1dccc35b-6206-47f3-b901-bafc5f17b8ad-kube-api-access-s5bg9\") pod \"router-default-566bb4455-kq2j4\" (UID: \"1dccc35b-6206-47f3-b901-bafc5f17b8ad\") " pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:48:35.100870 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.100837 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfz5j\" (UniqueName: \"kubernetes.io/projected/a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5-kube-api-access-bfz5j\") pod \"console-operator-7cd56b8dbc-fj4n7\" (UID: \"a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" Nov 02 22:48:35.101199 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.101181 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64dcq\" (UniqueName: \"kubernetes.io/projected/82b3b5e2-09f1-49c0-8d72-c85187918223-kube-api-access-64dcq\") pod \"network-metrics-daemon-465bf\" (UID: \"82b3b5e2-09f1-49c0-8d72-c85187918223\") " pod="openshift-multus/network-metrics-daemon-465bf" Nov 02 22:48:35.101608 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.101590 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjvmh\" (UniqueName: \"kubernetes.io/projected/ecdc26b5-be99-4928-8fff-203cc2c19ad6-kube-api-access-jjvmh\") pod \"kas-connection-checker-685895c7c9-25msk\" (UID: \"ecdc26b5-be99-4928-8fff-203cc2c19ad6\") " pod="kube-system/kas-connection-checker-685895c7c9-25msk" Nov 02 22:48:35.102210 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.102194 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njqr9\" (UniqueName: \"kubernetes.io/projected/cb204d6b-3758-4db4-bd68-0eacec87b805-kube-api-access-njqr9\") pod \"node-resolver-lhbwd\" (UID: \"cb204d6b-3758-4db4-bd68-0eacec87b805\") " pod="openshift-dns/node-resolver-lhbwd" Nov 02 22:48:35.103010 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.102991 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht6kh\" (UniqueName: \"kubernetes.io/projected/13a7b3bb-060c-4383-a864-0575834b221e-kube-api-access-ht6kh\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.103386 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.103365 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rn6f\" (UniqueName: \"kubernetes.io/projected/071262de-534b-4d01-bc20-784db9852f69-kube-api-access-7rn6f\") pod \"service-ca-operator-544d749c8c-xglvd\" (UID: \"071262de-534b-4d01-bc20-784db9852f69\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xglvd" Nov 02 22:48:35.103574 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.103556 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgmgr\" (UniqueName: \"kubernetes.io/projected/ff5ad4d1-4fd9-4374-b11c-78596c9cafa8-kube-api-access-cgmgr\") pod \"network-check-source-6f78d59f99-nbswd\" (UID: \"ff5ad4d1-4fd9-4374-b11c-78596c9cafa8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nbswd" Nov 02 22:48:35.103974 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.103954 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qw9l\" (UniqueName: \"kubernetes.io/projected/ae96a9c1-10a6-4050-bca0-6a2e7859d3c3-kube-api-access-9qw9l\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt\" (UID: \"ae96a9c1-10a6-4050-bca0-6a2e7859d3c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt" Nov 02 22:48:35.104434 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.104417 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnf2q\" (UniqueName: \"kubernetes.io/projected/a8d56c63-207c-42b3-8946-c993d0fde66b-kube-api-access-tnf2q\") pod \"kas-connection-checker-685895c7c9-5wnxz\" (UID: \"a8d56c63-207c-42b3-8946-c993d0fde66b\") " pod="kube-system/kas-connection-checker-685895c7c9-5wnxz" Nov 02 22:48:35.104577 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.104561 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6cmc\" (UniqueName: \"kubernetes.io/projected/2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc-kube-api-access-t6cmc\") pod \"cluster-samples-operator-6ddb6c8b94-rf5x6\" (UID: \"2e07d84f-46e6-4679-ad3d-7afa6d8cb8cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rf5x6" Nov 02 22:48:35.104991 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.104974 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gzjx\" (UniqueName: \"kubernetes.io/projected/457675c6-c725-44e5-a857-97aa4c17e655-kube-api-access-9gzjx\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:48:35.105323 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.105308 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t56h\" (UniqueName: \"kubernetes.io/projected/7321845c-9a95-415b-b6db-2bf983d77fa5-kube-api-access-2t56h\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.113464 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.113439 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t2t2\" (UniqueName: \"kubernetes.io/projected/fc560bbe-b31a-48e8-8416-dda72f49dae1-kube-api-access-5t2t2\") pod \"node-ca-mp88w\" (UID: \"fc560bbe-b31a-48e8-8416-dda72f49dae1\") " pod="openshift-image-registry/node-ca-mp88w" Nov 02 22:48:35.113523 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.113472 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvp6g\" (UniqueName: \"kubernetes.io/projected/fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c-kube-api-access-zvp6g\") pod \"insights-operator-7c4986d449-464jz\" (UID: \"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c\") " pod="openshift-insights/insights-operator-7c4986d449-464jz" Nov 02 22:48:35.145137 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145113 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-var-lib-cni-multus\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.145196 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145152 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-modprobe-d\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.145196 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145167 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-log-socket\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.145196 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145183 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-certificates\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:48:35.145336 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145207 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-bound-sa-token\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:48:35.145336 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145225 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-var-lib-cni-multus\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.145336 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145230 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/457675c6-c725-44e5-a857-97aa4c17e655-cnibin\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:48:35.145336 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145251 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-modprobe-d\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.145336 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145255 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-log-socket\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.145527 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145367 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/457675c6-c725-44e5-a857-97aa4c17e655-cnibin\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:48:35.145527 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145373 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc560bbe-b31a-48e8-8416-dda72f49dae1-host\") pod \"node-ca-mp88w\" (UID: \"fc560bbe-b31a-48e8-8416-dda72f49dae1\") " pod="openshift-image-registry/node-ca-mp88w" Nov 02 22:48:35.145527 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145399 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-certificates\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:48:35.145527 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145418 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-run-systemd\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.145527 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145449 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc560bbe-b31a-48e8-8416-dda72f49dae1-host\") pod \"node-ca-mp88w\" (UID: \"fc560bbe-b31a-48e8-8416-dda72f49dae1\") " pod="openshift-image-registry/node-ca-mp88w" Nov 02 22:48:35.145527 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145490 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-run-multus-certs\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.145527 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145509 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-run-systemd\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.145793 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145543 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-run-multus-certs\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.145793 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145564 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-systemd\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.145793 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145590 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-kubelet\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.145793 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145609 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-sysctl-conf\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.145793 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145636 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-kubelet\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.145793 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145646 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-cni-bin\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.145793 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145655 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-systemd\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.145793 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145674 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh6h5\" (UniqueName: \"kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-kube-api-access-gh6h5\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:48:35.145793 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145693 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwmn4\" (UniqueName: \"kubernetes.io/projected/215a42f8-881d-4cc4-b670-ea9f4aefb80b-kube-api-access-nwmn4\") pod \"ingress-canary-6jd4b\" (UID: \"215a42f8-881d-4cc4-b670-ea9f4aefb80b\") " pod="openshift-ingress-canary/ingress-canary-6jd4b" Nov 02 22:48:35.145793 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145713 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-multus-conf-dir\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.145793 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145718 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-cni-bin\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.145793 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145728 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72ef59a1-3734-47b7-9074-f683ca94dd32-metrics-tls\") pod \"dns-default-c8kg9\" (UID: \"72ef59a1-3734-47b7-9074-f683ca94dd32\") " pod="openshift-dns/dns-default-c8kg9" Nov 02 22:48:35.145793 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145730 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-sysctl-conf\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.145793 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145751 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-multus-conf-dir\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.145793 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145758 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-installation-pull-secrets\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145823 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qczmw\" (UniqueName: \"kubernetes.io/projected/72ef59a1-3734-47b7-9074-f683ca94dd32-kube-api-access-qczmw\") pod \"dns-default-c8kg9\" (UID: \"72ef59a1-3734-47b7-9074-f683ca94dd32\") " pod="openshift-dns/dns-default-c8kg9" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145931 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-host\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145958 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/457675c6-c725-44e5-a857-97aa4c17e655-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145970 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-installation-pull-secrets\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145981 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-sys-fs\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145996 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb204d6b-3758-4db4-bd68-0eacec87b805-hosts-file\") pod \"node-resolver-lhbwd\" (UID: \"cb204d6b-3758-4db4-bd68-0eacec87b805\") " pod="openshift-dns/node-resolver-lhbwd" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.145996 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-host\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146012 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-sys\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146034 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-hostroot\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146044 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb204d6b-3758-4db4-bd68-0eacec87b805-hosts-file\") pod \"node-resolver-lhbwd\" (UID: \"cb204d6b-3758-4db4-bd68-0eacec87b805\") " pod="openshift-dns/node-resolver-lhbwd" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146034 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-sys-fs\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146044 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/457675c6-c725-44e5-a857-97aa4c17e655-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146074 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-sys\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146073 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-run-netns\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146100 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-run-netns\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146083 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72ef59a1-3734-47b7-9074-f683ca94dd32-metrics-tls\") pod \"dns-default-c8kg9\" (UID: \"72ef59a1-3734-47b7-9074-f683ca94dd32\") " pod="openshift-dns/dns-default-c8kg9" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146081 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-hostroot\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146112 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ee0440bc-1614-4209-8744-6d197a019b5c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146160 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72ef59a1-3734-47b7-9074-f683ca94dd32-tmp-dir\") pod \"dns-default-c8kg9\" (UID: \"72ef59a1-3734-47b7-9074-f683ca94dd32\") " pod="openshift-dns/dns-default-c8kg9" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146177 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-image-registry-private-configuration\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146201 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-sysconfig\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146216 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-registration-dir\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146220 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72ef59a1-3734-47b7-9074-f683ca94dd32-tmp-dir\") pod \"dns-default-c8kg9\" (UID: \"72ef59a1-3734-47b7-9074-f683ca94dd32\") " pod="openshift-dns/dns-default-c8kg9" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146232 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee0440bc-1614-4209-8744-6d197a019b5c-data-volume\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146247 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-kubernetes\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146263 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/457675c6-c725-44e5-a857-97aa4c17e655-system-cni-dir\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146292 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-systemd-units\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146294 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-sysconfig\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146301 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-registration-dir\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146311 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ee0440bc-1614-4209-8744-6d197a019b5c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146318 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-var-lib-kubelet\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146324 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-kubernetes\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146345 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/457675c6-c725-44e5-a857-97aa4c17e655-system-cni-dir\") pod \"multus-additional-cni-plugins-t9rnl\" (UID: \"457675c6-c725-44e5-a857-97aa4c17e655\") " pod="openshift-multus/multus-additional-cni-plugins-t9rnl" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146352 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee0440bc-1614-4209-8744-6d197a019b5c-data-volume\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146363 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-image-registry-private-configuration\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146366 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-var-lib-kubelet\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146370 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-systemd-units\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146349 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-device-dir\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146393 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-device-dir\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146400 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-ca-trust-extracted\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146417 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/417d26e1-2a9d-41e3-8940-69626662a2f3-dbus\") pod \"global-pull-secret-syncer-wbxxp\" (UID: \"417d26e1-2a9d-41e3-8940-69626662a2f3\") " pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146431 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-var-lib-kubelet\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146455 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146467 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-var-lib-kubelet\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146470 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/417d26e1-2a9d-41e3-8940-69626662a2f3-dbus\") pod \"global-pull-secret-syncer-wbxxp\" (UID: \"417d26e1-2a9d-41e3-8940-69626662a2f3\") " pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146482 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-run-ovn\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146508 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146504 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ee0440bc-1614-4209-8744-6d197a019b5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:35.146505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146529 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-run-ovn\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146532 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-lib-modules\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146569 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-ca-trust-extracted\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146573 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-slash\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146600 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-var-lib-openvswitch\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146622 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-slash\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146604 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-lib-modules\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146626 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-etc-openvswitch\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146659 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-cni-netd\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146663 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-var-lib-openvswitch\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146664 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-etc-openvswitch\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146682 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-system-cni-dir\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146689 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ee0440bc-1614-4209-8744-6d197a019b5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146696 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-etc-selinux\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146714 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-trusted-ca\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146718 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-cni-netd\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146732 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwnw9\" (UniqueName: \"kubernetes.io/projected/ee0440bc-1614-4209-8744-6d197a019b5c-kube-api-access-jwnw9\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146741 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-system-cni-dir\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146749 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-etc-selinux\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146753 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-run-netns\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146836 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-run-ovn-kubernetes\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146852 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-run\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146866 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-cnibin\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146880 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-multus-socket-dir-parent\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146900 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-run-k8s-cni-cncf-io\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146901 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-run-ovn-kubernetes\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146915 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-socket-dir\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146924 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-run\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146935 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-node-log\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146937 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-cnibin\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146837 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-run-netns\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146958 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-run-k8s-cni-cncf-io\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146962 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146986 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146987 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/215a42f8-881d-4cc4-b670-ea9f4aefb80b-cert\") pod \"ingress-canary-6jd4b\" (UID: \"215a42f8-881d-4cc4-b670-ea9f4aefb80b\") " pod="openshift-ingress-canary/ingress-canary-6jd4b" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146990 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb045f55-e48d-4cac-88cf-90f90c225cbb-socket-dir\") pod \"aws-ebs-csi-driver-node-5hxsd\" (UID: \"fb045f55-e48d-4cac-88cf-90f90c225cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5hxsd" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146956 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-multus-socket-dir-parent\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.147012 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-run-openvswitch\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.147014 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-trusted-ca\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.146965 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-node-log\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.147044 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/472c7524-6a70-420c-aa56-7f3028a53d52-host-slash\") pod \"iptables-alerter-rw8v4\" (UID: \"472c7524-6a70-420c-aa56-7f3028a53d52\") " pod="openshift-network-operator/iptables-alerter-rw8v4" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.147047 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7321845c-9a95-415b-b6db-2bf983d77fa5-run-openvswitch\") pod \"ovnkube-node-pl8m7\" (UID: \"7321845c-9a95-415b-b6db-2bf983d77fa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.147152 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/72ef59a1-3734-47b7-9074-f683ca94dd32-config-volume\") pod \"dns-default-c8kg9\" (UID: \"72ef59a1-3734-47b7-9074-f683ca94dd32\") " pod="openshift-dns/dns-default-c8kg9" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.147149 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/472c7524-6a70-420c-aa56-7f3028a53d52-host-slash\") pod \"iptables-alerter-rw8v4\" (UID: \"472c7524-6a70-420c-aa56-7f3028a53d52\") " pod="openshift-network-operator/iptables-alerter-rw8v4" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.147201 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-multus-cni-dir\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.147216 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/215a42f8-881d-4cc4-b670-ea9f4aefb80b-cert\") pod \"ingress-canary-6jd4b\" (UID: \"215a42f8-881d-4cc4-b670-ea9f4aefb80b\") " pod="openshift-ingress-canary/ingress-canary-6jd4b" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.147232 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-etc-kubernetes\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.147252 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/417d26e1-2a9d-41e3-8940-69626662a2f3-kubelet-config\") pod \"global-pull-secret-syncer-wbxxp\" (UID: \"417d26e1-2a9d-41e3-8940-69626662a2f3\") " pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.147268 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-tls\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.147265 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-multus-cni-dir\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.147274 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/72ef59a1-3734-47b7-9074-f683ca94dd32-config-volume\") pod \"dns-default-c8kg9\" (UID: \"72ef59a1-3734-47b7-9074-f683ca94dd32\") " pod="openshift-dns/dns-default-c8kg9" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.147284 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8t9t\" (UniqueName: \"kubernetes.io/projected/06f6f89f-b8f9-45ec-b50a-f3407bfe7ef8-kube-api-access-b8t9t\") pod \"migrator-66df5d496b-ttdqt\" (UID: \"06f6f89f-b8f9-45ec-b50a-f3407bfe7ef8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ttdqt" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.147305 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-etc-kubernetes\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.147319 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/417d26e1-2a9d-41e3-8940-69626662a2f3-kubelet-config\") pod \"global-pull-secret-syncer-wbxxp\" (UID: \"417d26e1-2a9d-41e3-8940-69626662a2f3\") " pod="kube-system/global-pull-secret-syncer-wbxxp" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.147331 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-sysctl-d\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.147347 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-var-lib-cni-bin\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.147364 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ee0440bc-1614-4209-8744-6d197a019b5c-crio-socket\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.147413 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-tls\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.147441 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26b7328d-dacf-44c2-995b-306e3ef9cf31-host-var-lib-cni-bin\") pod \"multus-mcvpd\" (UID: \"26b7328d-dacf-44c2-995b-306e3ef9cf31\") " pod="openshift-multus/multus-mcvpd" Nov 02 22:48:35.148184 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.147477 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/13a7b3bb-060c-4383-a864-0575834b221e-etc-sysctl-d\") pod \"tuned-2kt76\" (UID: \"13a7b3bb-060c-4383-a864-0575834b221e\") " pod="openshift-cluster-node-tuning-operator/tuned-2kt76" Nov 02 22:48:35.203528 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.203500 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-bound-sa-token\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:48:35.203976 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.203954 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qczmw\" (UniqueName: \"kubernetes.io/projected/72ef59a1-3734-47b7-9074-f683ca94dd32-kube-api-access-qczmw\") pod \"dns-default-c8kg9\" (UID: \"72ef59a1-3734-47b7-9074-f683ca94dd32\") " pod="openshift-dns/dns-default-c8kg9" Nov 02 22:48:35.204047 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.204012 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwmn4\" (UniqueName: \"kubernetes.io/projected/215a42f8-881d-4cc4-b670-ea9f4aefb80b-kube-api-access-nwmn4\") pod \"ingress-canary-6jd4b\" (UID: \"215a42f8-881d-4cc4-b670-ea9f4aefb80b\") " pod="openshift-ingress-canary/ingress-canary-6jd4b" Nov 02 22:48:35.204370 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.204350 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh6h5\" (UniqueName: \"kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-kube-api-access-gh6h5\") pod \"image-registry-bc8cbc8c-hghqr\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:48:35.204555 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.204540 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwnw9\" (UniqueName: \"kubernetes.io/projected/ee0440bc-1614-4209-8744-6d197a019b5c-kube-api-access-jwnw9\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:35.204738 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.204721 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8t9t\" (UniqueName: \"kubernetes.io/projected/06f6f89f-b8f9-45ec-b50a-f3407bfe7ef8-kube-api-access-b8t9t\") pod \"migrator-66df5d496b-ttdqt\" (UID: \"06f6f89f-b8f9-45ec-b50a-f3407bfe7ef8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ttdqt" Nov 02 22:48:35.231940 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.231916 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:48:35.232759 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.232745 5727 scope.go:122] "RemoveContainer" containerID="de773a7c0efbc6b4377ddff49679469091bfed1e165b3ad6bfdfa9bd6760db62" Nov 02 22:48:35.247737 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.247716 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ee0440bc-1614-4209-8744-6d197a019b5c-crio-socket\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:35.247810 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.247794 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ee0440bc-1614-4209-8744-6d197a019b5c-crio-socket\") pod \"insights-runtime-extractor-qlfjm\" (UID: \"ee0440bc-1614-4209-8744-6d197a019b5c\") " pod="openshift-insights/insights-runtime-extractor-qlfjm" Nov 02 22:48:35.546258 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.546185 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2qpkq" Nov 02 22:48:35.935192 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.935165 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-c8kg9" Nov 02 22:48:35.952425 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.952402 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 22:48:35.952549 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.952493 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" event={"ID":"a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5","Type":"ContainerStarted","Data":"aaa5f842d3e077d822ab579373839868778b6447d89d93cc9fa9cc423245142b"} Nov 02 22:48:35.954109 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.954087 5727 generic.go:356] "Generic (PLEG): container finished" podID="fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c" containerID="7239b669767a2db3a54679623bc6fa6aaae096af8e98695ab41a369d27ebc3a3" exitCode=0 Nov 02 22:48:35.954233 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.954152 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-464jz" event={"ID":"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c","Type":"ContainerDied","Data":"7239b669767a2db3a54679623bc6fa6aaae096af8e98695ab41a369d27ebc3a3"} Nov 02 22:48:35.954472 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.954457 5727 scope.go:122] "RemoveContainer" containerID="7239b669767a2db3a54679623bc6fa6aaae096af8e98695ab41a369d27ebc3a3" Nov 02 22:48:35.955925 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.955904 5727 generic.go:356] "Generic (PLEG): container finished" podID="ae96a9c1-10a6-4050-bca0-6a2e7859d3c3" containerID="ea853af35c2d3f487a8ee38b3a66286d0b5854ad7e11250bb21104980d781f30" exitCode=0 Nov 02 22:48:35.956046 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.955990 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt" event={"ID":"ae96a9c1-10a6-4050-bca0-6a2e7859d3c3","Type":"ContainerDied","Data":"ea853af35c2d3f487a8ee38b3a66286d0b5854ad7e11250bb21104980d781f30"} Nov 02 22:48:35.956395 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.956377 5727 scope.go:122] "RemoveContainer" containerID="ea853af35c2d3f487a8ee38b3a66286d0b5854ad7e11250bb21104980d781f30" Nov 02 22:48:35.976100 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:35.976076 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" Nov 02 22:48:36.113367 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:36.113340 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-fj4n7" Nov 02 22:48:36.233182 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:36.233095 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:48:36.963110 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:36.963079 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8rlqt" event={"ID":"ae96a9c1-10a6-4050-bca0-6a2e7859d3c3","Type":"ContainerStarted","Data":"a1a825059c37dc7339f9bff5d671fb29a0f8386a31a09dc4341779df3459a0c3"} Nov 02 22:48:36.964780 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:36.964755 5727 generic.go:356] "Generic (PLEG): container finished" podID="071262de-534b-4d01-bc20-784db9852f69" containerID="4fd81f46795417b11a87ac4342bd040579866e3f757a8893efa1d247f9d07056" exitCode=0 Nov 02 22:48:36.964897 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:36.964812 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xglvd" event={"ID":"071262de-534b-4d01-bc20-784db9852f69","Type":"ContainerDied","Data":"4fd81f46795417b11a87ac4342bd040579866e3f757a8893efa1d247f9d07056"} Nov 02 22:48:36.965149 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:36.965117 5727 scope.go:122] "RemoveContainer" containerID="4fd81f46795417b11a87ac4342bd040579866e3f757a8893efa1d247f9d07056" Nov 02 22:48:36.967014 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:36.966987 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-464jz" event={"ID":"fb8c24bf-0fc9-4d8b-84a6-2fc5d652950c","Type":"ContainerStarted","Data":"6af4856a8ce40bb06fce1a055fe0ee08cda9c3921c1b4938d3db87f974a6ec30"} Nov 02 22:48:37.972763 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:37.972738 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:37.973824 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:37.973797 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xglvd" event={"ID":"071262de-534b-4d01-bc20-784db9852f69","Type":"ContainerStarted","Data":"dc14c6911454c5350ce16e1b22311c0f72edf636cf8b1d3b2e825ad02fe19b87"} Nov 02 22:48:39.302815 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:39.302782 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:48:39.303218 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:39.303029 5727 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 22:48:40.007761 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:40.007723 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c8kg9_72ef59a1-3734-47b7-9074-f683ca94dd32/dns/0.log" Nov 02 22:48:40.111359 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:40.111324 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c8kg9_72ef59a1-3734-47b7-9074-f683ca94dd32/kube-rbac-proxy/0.log" Nov 02 22:48:41.077922 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:41.077886 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:48:41.078324 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:41.078032 5727 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 22:48:41.079217 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:41.079196 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-566bb4455-kq2j4" Nov 02 22:48:41.119266 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:41.119243 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lhbwd_cb204d6b-3758-4db4-bd68-0eacec87b805/dns-node-resolver/0.log" Nov 02 22:48:41.320419 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:41.320387 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-566bb4455-kq2j4_1dccc35b-6206-47f3-b901-bafc5f17b8ad/router/0.log" Nov 02 22:48:41.519513 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:41.519426 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6jd4b_215a42f8-881d-4cc4-b670-ea9f4aefb80b/serve-healthcheck-canary/0.log" Nov 02 22:48:41.958477 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:41.958450 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-c8kg9" Nov 02 22:48:42.868893 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:42.868864 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-r5bbz"] Nov 02 22:48:42.871151 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:42.871135 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r5bbz" Nov 02 22:48:42.901031 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:42.901005 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f0d480ea-7dd1-4e56-93f4-b2007ee9ba54-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r5bbz\" (UID: \"f0d480ea-7dd1-4e56-93f4-b2007ee9ba54\") " pod="openshift-monitoring/node-exporter-r5bbz" Nov 02 22:48:42.901174 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:42.901041 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0d480ea-7dd1-4e56-93f4-b2007ee9ba54-sys\") pod \"node-exporter-r5bbz\" (UID: \"f0d480ea-7dd1-4e56-93f4-b2007ee9ba54\") " pod="openshift-monitoring/node-exporter-r5bbz" Nov 02 22:48:42.901174 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:42.901066 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f0d480ea-7dd1-4e56-93f4-b2007ee9ba54-node-exporter-textfile\") pod \"node-exporter-r5bbz\" (UID: \"f0d480ea-7dd1-4e56-93f4-b2007ee9ba54\") " pod="openshift-monitoring/node-exporter-r5bbz" Nov 02 22:48:42.901174 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:42.901144 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f0d480ea-7dd1-4e56-93f4-b2007ee9ba54-node-exporter-tls\") pod \"node-exporter-r5bbz\" (UID: \"f0d480ea-7dd1-4e56-93f4-b2007ee9ba54\") " pod="openshift-monitoring/node-exporter-r5bbz" Nov 02 22:48:42.901363 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:42.901227 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f0d480ea-7dd1-4e56-93f4-b2007ee9ba54-root\") pod \"node-exporter-r5bbz\" (UID: \"f0d480ea-7dd1-4e56-93f4-b2007ee9ba54\") " pod="openshift-monitoring/node-exporter-r5bbz" Nov 02 22:48:42.901363 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:42.901330 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f0d480ea-7dd1-4e56-93f4-b2007ee9ba54-node-exporter-accelerators-collector-config\") pod \"node-exporter-r5bbz\" (UID: \"f0d480ea-7dd1-4e56-93f4-b2007ee9ba54\") " pod="openshift-monitoring/node-exporter-r5bbz" Nov 02 22:48:42.901363 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:42.901360 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f0d480ea-7dd1-4e56-93f4-b2007ee9ba54-node-exporter-wtmp\") pod \"node-exporter-r5bbz\" (UID: \"f0d480ea-7dd1-4e56-93f4-b2007ee9ba54\") " pod="openshift-monitoring/node-exporter-r5bbz" Nov 02 22:48:42.901494 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:42.901376 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvs87\" (UniqueName: \"kubernetes.io/projected/f0d480ea-7dd1-4e56-93f4-b2007ee9ba54-kube-api-access-cvs87\") pod \"node-exporter-r5bbz\" (UID: \"f0d480ea-7dd1-4e56-93f4-b2007ee9ba54\") " pod="openshift-monitoring/node-exporter-r5bbz" Nov 02 22:48:42.901494 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:42.901431 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0d480ea-7dd1-4e56-93f4-b2007ee9ba54-metrics-client-ca\") pod \"node-exporter-r5bbz\" (UID: \"f0d480ea-7dd1-4e56-93f4-b2007ee9ba54\") " pod="openshift-monitoring/node-exporter-r5bbz" Nov 02 22:48:42.923188 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:42.923150 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 22:48:42.972682 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:42.972651 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 22:48:42.972775 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:42.972710 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 22:48:42.972775 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:42.972742 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 22:48:42.972834 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:42.972813 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-scv9s\"" Nov 02 22:48:43.002499 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:43.002470 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f0d480ea-7dd1-4e56-93f4-b2007ee9ba54-node-exporter-accelerators-collector-config\") pod \"node-exporter-r5bbz\" (UID: \"f0d480ea-7dd1-4e56-93f4-b2007ee9ba54\") " pod="openshift-monitoring/node-exporter-r5bbz" Nov 02 22:48:43.002633 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:43.002505 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f0d480ea-7dd1-4e56-93f4-b2007ee9ba54-node-exporter-wtmp\") pod \"node-exporter-r5bbz\" (UID: \"f0d480ea-7dd1-4e56-93f4-b2007ee9ba54\") " pod="openshift-monitoring/node-exporter-r5bbz" Nov 02 22:48:43.002633 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:43.002522 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvs87\" (UniqueName: \"kubernetes.io/projected/f0d480ea-7dd1-4e56-93f4-b2007ee9ba54-kube-api-access-cvs87\") pod \"node-exporter-r5bbz\" (UID: \"f0d480ea-7dd1-4e56-93f4-b2007ee9ba54\") " pod="openshift-monitoring/node-exporter-r5bbz" Nov 02 22:48:43.002633 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:43.002545 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0d480ea-7dd1-4e56-93f4-b2007ee9ba54-metrics-client-ca\") pod \"node-exporter-r5bbz\" (UID: \"f0d480ea-7dd1-4e56-93f4-b2007ee9ba54\") " pod="openshift-monitoring/node-exporter-r5bbz" Nov 02 22:48:43.002633 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:43.002565 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f0d480ea-7dd1-4e56-93f4-b2007ee9ba54-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r5bbz\" (UID: \"f0d480ea-7dd1-4e56-93f4-b2007ee9ba54\") " pod="openshift-monitoring/node-exporter-r5bbz" Nov 02 22:48:43.002633 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:43.002592 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0d480ea-7dd1-4e56-93f4-b2007ee9ba54-sys\") pod \"node-exporter-r5bbz\" (UID: \"f0d480ea-7dd1-4e56-93f4-b2007ee9ba54\") " pod="openshift-monitoring/node-exporter-r5bbz" Nov 02 22:48:43.002874 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:43.002642 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0d480ea-7dd1-4e56-93f4-b2007ee9ba54-sys\") pod \"node-exporter-r5bbz\" (UID: \"f0d480ea-7dd1-4e56-93f4-b2007ee9ba54\") " pod="openshift-monitoring/node-exporter-r5bbz" Nov 02 22:48:43.002874 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:43.002671 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f0d480ea-7dd1-4e56-93f4-b2007ee9ba54-node-exporter-textfile\") pod \"node-exporter-r5bbz\" (UID: \"f0d480ea-7dd1-4e56-93f4-b2007ee9ba54\") " pod="openshift-monitoring/node-exporter-r5bbz" Nov 02 22:48:43.002874 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:43.002679 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f0d480ea-7dd1-4e56-93f4-b2007ee9ba54-node-exporter-wtmp\") pod \"node-exporter-r5bbz\" (UID: \"f0d480ea-7dd1-4e56-93f4-b2007ee9ba54\") " pod="openshift-monitoring/node-exporter-r5bbz" Nov 02 22:48:43.002874 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:43.002706 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f0d480ea-7dd1-4e56-93f4-b2007ee9ba54-node-exporter-tls\") pod \"node-exporter-r5bbz\" (UID: \"f0d480ea-7dd1-4e56-93f4-b2007ee9ba54\") " pod="openshift-monitoring/node-exporter-r5bbz" Nov 02 22:48:43.002874 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:43.002729 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f0d480ea-7dd1-4e56-93f4-b2007ee9ba54-root\") pod \"node-exporter-r5bbz\" (UID: \"f0d480ea-7dd1-4e56-93f4-b2007ee9ba54\") " pod="openshift-monitoring/node-exporter-r5bbz" Nov 02 22:48:43.002874 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:43.002802 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f0d480ea-7dd1-4e56-93f4-b2007ee9ba54-root\") pod \"node-exporter-r5bbz\" (UID: \"f0d480ea-7dd1-4e56-93f4-b2007ee9ba54\") " pod="openshift-monitoring/node-exporter-r5bbz" Nov 02 22:48:43.003181 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:43.002999 5727 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 22:48:43.003181 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:43.003025 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f0d480ea-7dd1-4e56-93f4-b2007ee9ba54-node-exporter-textfile\") pod \"node-exporter-r5bbz\" (UID: \"f0d480ea-7dd1-4e56-93f4-b2007ee9ba54\") " pod="openshift-monitoring/node-exporter-r5bbz" Nov 02 22:48:43.003288 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:43.003241 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0d480ea-7dd1-4e56-93f4-b2007ee9ba54-metrics-client-ca\") pod \"node-exporter-r5bbz\" (UID: \"f0d480ea-7dd1-4e56-93f4-b2007ee9ba54\") " pod="openshift-monitoring/node-exporter-r5bbz" Nov 02 22:48:43.003288 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:43.003247 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f0d480ea-7dd1-4e56-93f4-b2007ee9ba54-node-exporter-accelerators-collector-config\") pod \"node-exporter-r5bbz\" (UID: \"f0d480ea-7dd1-4e56-93f4-b2007ee9ba54\") " pod="openshift-monitoring/node-exporter-r5bbz" Nov 02 22:48:43.005202 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:43.005185 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f0d480ea-7dd1-4e56-93f4-b2007ee9ba54-node-exporter-tls\") pod \"node-exporter-r5bbz\" (UID: \"f0d480ea-7dd1-4e56-93f4-b2007ee9ba54\") " pod="openshift-monitoring/node-exporter-r5bbz" Nov 02 22:48:43.005246 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:43.005227 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f0d480ea-7dd1-4e56-93f4-b2007ee9ba54-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r5bbz\" (UID: \"f0d480ea-7dd1-4e56-93f4-b2007ee9ba54\") " pod="openshift-monitoring/node-exporter-r5bbz" Nov 02 22:48:43.033613 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:43.033592 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:48:43.037498 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:43.037457 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:48:43.062805 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:43.062773 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvs87\" (UniqueName: \"kubernetes.io/projected/f0d480ea-7dd1-4e56-93f4-b2007ee9ba54-kube-api-access-cvs87\") pod \"node-exporter-r5bbz\" (UID: \"f0d480ea-7dd1-4e56-93f4-b2007ee9ba54\") " pod="openshift-monitoring/node-exporter-r5bbz" Nov 02 22:48:43.183767 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:43.183694 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r5bbz" Nov 02 22:48:43.195726 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:48:43.195690 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d480ea_7dd1_4e56_93f4_b2007ee9ba54.slice/crio-1e37fb3b8bd2516db100d34a2db64f32d1890042ce7f221c0840e4dbbf1526b6 WatchSource:0}: Error finding container 1e37fb3b8bd2516db100d34a2db64f32d1890042ce7f221c0840e4dbbf1526b6: Status 404 returned error can't find the container with id 1e37fb3b8bd2516db100d34a2db64f32d1890042ce7f221c0840e4dbbf1526b6 Nov 02 22:48:43.197672 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:43.197650 5727 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:48:43.996182 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:43.996144 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r5bbz" event={"ID":"f0d480ea-7dd1-4e56-93f4-b2007ee9ba54","Type":"ContainerStarted","Data":"1e37fb3b8bd2516db100d34a2db64f32d1890042ce7f221c0840e4dbbf1526b6"} Nov 02 22:48:45.001433 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:45.001356 5727 generic.go:356] "Generic (PLEG): container finished" podID="f0d480ea-7dd1-4e56-93f4-b2007ee9ba54" containerID="f6480b4d2bfe45f4ca1db4b270ed601928cd52f1bee5ea5fdc00c3998cf113c8" exitCode=0 Nov 02 22:48:45.001433 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:45.001393 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r5bbz" event={"ID":"f0d480ea-7dd1-4e56-93f4-b2007ee9ba54","Type":"ContainerDied","Data":"f6480b4d2bfe45f4ca1db4b270ed601928cd52f1bee5ea5fdc00c3998cf113c8"} Nov 02 22:48:46.007044 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:46.007012 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r5bbz" event={"ID":"f0d480ea-7dd1-4e56-93f4-b2007ee9ba54","Type":"ContainerStarted","Data":"1f8f0d08ad05b8c778fb8c79fe77d3a675d9afba7d367e06f9eda6ced3854fe8"} Nov 02 22:48:46.007572 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:46.007065 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r5bbz" event={"ID":"f0d480ea-7dd1-4e56-93f4-b2007ee9ba54","Type":"ContainerStarted","Data":"47dd6f09dea32f335d0c7549055dac09358c4d17a214d86f6d83106b7a66bded"} Nov 02 22:48:46.125279 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:46.125209 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-r5bbz" podStartSLOduration=2.529869351 podStartE2EDuration="4.125194502s" totalImagesPullingTime="1.519011627s" totalInitContainerRuntime="76.313524ms" isStatefulPod=true podCreationTimestamp="2026-11-02 22:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:46.124924602 +0000 UTC m=+12.363422266" watchObservedRunningTime="2026-11-02 22:48:46.125194502 +0000 UTC m=+12.363692161" Nov 02 22:48:54.393146 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:54.393104 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-bc8cbc8c-hghqr"] Nov 02 22:48:57.572305 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:48:57.572273 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-r5lnw" Nov 02 22:48:57.844645 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:48:57.844574 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 22:49:00.264475 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:00.263267 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pl8m7" Nov 02 22:49:19.413258 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:19.413187 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" podUID="2d578e40-60d8-4ebd-b0b5-2f5c274c7903" containerName="registry" containerID="cri-o://7d50faaaa721bb4fbadb50f25d90ff8b0b97daecec94622ee6a6c11a4fcc8ab6" gracePeriod=30 Nov 02 22:49:19.667569 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:19.667503 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:49:19.793655 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:19.793624 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-certificates\") pod \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " Nov 02 22:49:19.793655 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:19.793658 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-installation-pull-secrets\") pod \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " Nov 02 22:49:19.793906 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:19.793688 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-image-registry-private-configuration\") pod \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " Nov 02 22:49:19.793906 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:19.793723 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-bound-sa-token\") pod \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " Nov 02 22:49:19.793906 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:19.793748 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-kube-api-access-gh6h5\" (UniqueName: \"kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-kube-api-access-gh6h5\") pod \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " Nov 02 22:49:19.793906 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:19.793768 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-ca-trust-extracted\") pod \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " Nov 02 22:49:19.793906 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:19.793829 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-tls\" (UniqueName: \"kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-tls\") pod \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " Nov 02 22:49:19.793906 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:19.793863 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-trusted-ca\") pod \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\" (UID: \"2d578e40-60d8-4ebd-b0b5-2f5c274c7903\") " Nov 02 22:49:19.794292 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:19.794072 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-certificates" pod "2d578e40-60d8-4ebd-b0b5-2f5c274c7903" (UID: "2d578e40-60d8-4ebd-b0b5-2f5c274c7903"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:49:19.795111 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:19.795057 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-trusted-ca" pod "2d578e40-60d8-4ebd-b0b5-2f5c274c7903" (UID: "2d578e40-60d8-4ebd-b0b5-2f5c274c7903"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:49:19.796989 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:19.796944 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-bound-sa-token" pod "2d578e40-60d8-4ebd-b0b5-2f5c274c7903" (UID: "2d578e40-60d8-4ebd-b0b5-2f5c274c7903"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:49:19.797101 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:19.796985 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-installation-pull-secrets" pod "2d578e40-60d8-4ebd-b0b5-2f5c274c7903" (UID: "2d578e40-60d8-4ebd-b0b5-2f5c274c7903"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:49:19.800604 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:19.798667 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-image-registry-private-configuration" pod "2d578e40-60d8-4ebd-b0b5-2f5c274c7903" (UID: "2d578e40-60d8-4ebd-b0b5-2f5c274c7903"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:49:19.801171 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:19.800729 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-kube-api-access-gh6h5" pod "2d578e40-60d8-4ebd-b0b5-2f5c274c7903" (UID: "2d578e40-60d8-4ebd-b0b5-2f5c274c7903"). InnerVolumeSpecName "kube-api-access-gh6h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:49:19.802581 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:19.802556 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-tls" pod "2d578e40-60d8-4ebd-b0b5-2f5c274c7903" (UID: "2d578e40-60d8-4ebd-b0b5-2f5c274c7903"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:49:19.805056 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:19.805031 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-ca-trust-extracted" pod "2d578e40-60d8-4ebd-b0b5-2f5c274c7903" (UID: "2d578e40-60d8-4ebd-b0b5-2f5c274c7903"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:49:19.895277 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:19.895250 5727 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-trusted-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:19.895277 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:19.895275 5727 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-certificates\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:19.895462 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:19.895288 5727 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-installation-pull-secrets\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:19.895462 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:19.895304 5727 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-image-registry-private-configuration\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:19.895462 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:19.895314 5727 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-bound-sa-token\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:19.895462 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:19.895322 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh6h5\" (UniqueName: \"kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-kube-api-access-gh6h5\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:19.895462 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:19.895331 5727 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-ca-trust-extracted\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:19.895462 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:19.895339 5727 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d578e40-60d8-4ebd-b0b5-2f5c274c7903-registry-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:20.126072 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:20.126041 5727 generic.go:356] "Generic (PLEG): container finished" podID="2d578e40-60d8-4ebd-b0b5-2f5c274c7903" containerID="7d50faaaa721bb4fbadb50f25d90ff8b0b97daecec94622ee6a6c11a4fcc8ab6" exitCode=0 Nov 02 22:49:20.126249 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:20.126108 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" Nov 02 22:49:20.126249 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:20.126154 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" event={"ID":"2d578e40-60d8-4ebd-b0b5-2f5c274c7903","Type":"ContainerDied","Data":"7d50faaaa721bb4fbadb50f25d90ff8b0b97daecec94622ee6a6c11a4fcc8ab6"} Nov 02 22:49:20.126249 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:20.126210 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bc8cbc8c-hghqr" event={"ID":"2d578e40-60d8-4ebd-b0b5-2f5c274c7903","Type":"ContainerDied","Data":"4fba275614bd2ddaf44e9966d406d568f3b834031d711e5499040da2ca9c527d"} Nov 02 22:49:20.126249 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:20.126237 5727 scope.go:122] "RemoveContainer" containerID="7d50faaaa721bb4fbadb50f25d90ff8b0b97daecec94622ee6a6c11a4fcc8ab6" Nov 02 22:49:20.136224 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:20.136203 5727 scope.go:122] "RemoveContainer" containerID="7d50faaaa721bb4fbadb50f25d90ff8b0b97daecec94622ee6a6c11a4fcc8ab6" Nov 02 22:49:20.136533 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:49:20.136512 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d50faaaa721bb4fbadb50f25d90ff8b0b97daecec94622ee6a6c11a4fcc8ab6\": container with ID starting with 7d50faaaa721bb4fbadb50f25d90ff8b0b97daecec94622ee6a6c11a4fcc8ab6 not found: ID does not exist" containerID="7d50faaaa721bb4fbadb50f25d90ff8b0b97daecec94622ee6a6c11a4fcc8ab6" Nov 02 22:49:20.136580 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:20.136546 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d50faaaa721bb4fbadb50f25d90ff8b0b97daecec94622ee6a6c11a4fcc8ab6"} err="failed to get container status \"7d50faaaa721bb4fbadb50f25d90ff8b0b97daecec94622ee6a6c11a4fcc8ab6\": rpc error: code = NotFound desc = could not find container \"7d50faaaa721bb4fbadb50f25d90ff8b0b97daecec94622ee6a6c11a4fcc8ab6\": container with ID starting with 7d50faaaa721bb4fbadb50f25d90ff8b0b97daecec94622ee6a6c11a4fcc8ab6 not found: ID does not exist" Nov 02 22:49:20.288606 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:20.288579 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-bc8cbc8c-hghqr"] Nov 02 22:49:20.338524 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:20.338503 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-bc8cbc8c-hghqr"] Nov 02 22:49:21.879696 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:49:21.879662 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d578e40-60d8-4ebd-b0b5-2f5c274c7903" path="/var/lib/kubelet/pods/2d578e40-60d8-4ebd-b0b5-2f5c274c7903/volumes" Nov 02 22:49:27.843866 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:49:27.843829 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 22:49:57.843887 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:49:57.843842 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 22:50:27.845436 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:50:27.845389 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 22:50:57.844761 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:50:57.844721 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 22:51:27.844195 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:51:27.844154 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 22:51:57.844102 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:51:57.844065 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 22:52:27.843873 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:52:27.843836 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 22:52:57.800708 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:57.800668 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7ngb8"] Nov 02 22:52:57.801178 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:57.800974 5727 cpu_manager.go:414] "containerMap: removing container" podUID="2d578e40-60d8-4ebd-b0b5-2f5c274c7903" containerName="registry" Nov 02 22:52:57.801178 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:57.800985 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d578e40-60d8-4ebd-b0b5-2f5c274c7903" containerName="registry" Nov 02 22:52:57.801178 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:57.801048 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d578e40-60d8-4ebd-b0b5-2f5c274c7903" containerName="registry" Nov 02 22:52:57.803011 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:57.802994 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7ngb8" Nov 02 22:52:57.845385 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:52:57.845341 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 22:52:57.904696 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:57.904660 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 22:52:57.904696 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:57.904680 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 22:52:57.904696 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:57.904680 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-dwwz4\"" Nov 02 22:52:57.905605 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:57.905581 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7ngb8"] Nov 02 22:52:57.967042 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:57.967012 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/495baab4-1e67-4202-8cb1-ab002c5dd19a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7ngb8\" (UID: \"495baab4-1e67-4202-8cb1-ab002c5dd19a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7ngb8" Nov 02 22:52:57.967233 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:57.967048 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/495baab4-1e67-4202-8cb1-ab002c5dd19a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7ngb8\" (UID: \"495baab4-1e67-4202-8cb1-ab002c5dd19a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7ngb8" Nov 02 22:52:57.967233 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:57.967173 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvt5n\" (UniqueName: \"kubernetes.io/projected/495baab4-1e67-4202-8cb1-ab002c5dd19a-kube-api-access-qvt5n\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7ngb8\" (UID: \"495baab4-1e67-4202-8cb1-ab002c5dd19a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7ngb8" Nov 02 22:52:58.068137 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.068087 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/495baab4-1e67-4202-8cb1-ab002c5dd19a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7ngb8\" (UID: \"495baab4-1e67-4202-8cb1-ab002c5dd19a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7ngb8" Nov 02 22:52:58.068314 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.068176 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/495baab4-1e67-4202-8cb1-ab002c5dd19a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7ngb8\" (UID: \"495baab4-1e67-4202-8cb1-ab002c5dd19a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7ngb8" Nov 02 22:52:58.068314 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.068221 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvt5n\" (UniqueName: \"kubernetes.io/projected/495baab4-1e67-4202-8cb1-ab002c5dd19a-kube-api-access-qvt5n\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7ngb8\" (UID: \"495baab4-1e67-4202-8cb1-ab002c5dd19a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7ngb8" Nov 02 22:52:58.068491 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.068471 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/495baab4-1e67-4202-8cb1-ab002c5dd19a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7ngb8\" (UID: \"495baab4-1e67-4202-8cb1-ab002c5dd19a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7ngb8" Nov 02 22:52:58.068529 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.068501 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/495baab4-1e67-4202-8cb1-ab002c5dd19a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7ngb8\" (UID: \"495baab4-1e67-4202-8cb1-ab002c5dd19a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7ngb8" Nov 02 22:52:58.129318 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.129279 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvt5n\" (UniqueName: \"kubernetes.io/projected/495baab4-1e67-4202-8cb1-ab002c5dd19a-kube-api-access-qvt5n\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7ngb8\" (UID: \"495baab4-1e67-4202-8cb1-ab002c5dd19a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7ngb8" Nov 02 22:52:58.355361 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.355292 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67265n"] Nov 02 22:52:58.357668 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.357653 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67265n" Nov 02 22:52:58.409741 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.409709 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-4whqt\"" Nov 02 22:52:58.416591 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.416563 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67265n"] Nov 02 22:52:58.420887 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.420862 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7ngb8" Nov 02 22:52:58.453494 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.453458 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv9n54"] Nov 02 22:52:58.456046 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.456019 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv9n54" Nov 02 22:52:58.472320 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.472187 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67265n\" (UID: \"9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67265n" Nov 02 22:52:58.472320 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.472277 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67265n\" (UID: \"9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67265n" Nov 02 22:52:58.472494 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.472343 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5xk7\" (UniqueName: \"kubernetes.io/projected/9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c-kube-api-access-f5xk7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67265n\" (UID: \"9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67265n" Nov 02 22:52:58.509598 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.509562 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-tqsch\"" Nov 02 22:52:58.527332 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.527299 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv9n54"] Nov 02 22:52:58.553575 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.553546 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72f2rr"] Nov 02 22:52:58.555854 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.555837 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72f2rr" Nov 02 22:52:58.573236 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.573211 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67265n\" (UID: \"9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67265n" Nov 02 22:52:58.573346 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.573257 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67265n\" (UID: \"9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67265n" Nov 02 22:52:58.573346 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.573280 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q2l7\" (UniqueName: \"kubernetes.io/projected/6ba77a61-d033-4b7f-accf-2d842e49d934-kube-api-access-7q2l7\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv9n54\" (UID: \"6ba77a61-d033-4b7f-accf-2d842e49d934\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv9n54" Nov 02 22:52:58.573346 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.573308 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ba77a61-d033-4b7f-accf-2d842e49d934-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv9n54\" (UID: \"6ba77a61-d033-4b7f-accf-2d842e49d934\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv9n54" Nov 02 22:52:58.573346 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.573331 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5xk7\" (UniqueName: \"kubernetes.io/projected/9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c-kube-api-access-f5xk7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67265n\" (UID: \"9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67265n" Nov 02 22:52:58.573515 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.573353 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ba77a61-d033-4b7f-accf-2d842e49d934-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv9n54\" (UID: \"6ba77a61-d033-4b7f-accf-2d842e49d934\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv9n54" Nov 02 22:52:58.573653 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.573632 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67265n\" (UID: \"9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67265n" Nov 02 22:52:58.573690 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.573659 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67265n\" (UID: \"9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67265n" Nov 02 22:52:58.614237 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.614161 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-w6pcr\"" Nov 02 22:52:58.617148 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.617102 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72f2rr"] Nov 02 22:52:58.634284 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.634251 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5xk7\" (UniqueName: \"kubernetes.io/projected/9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c-kube-api-access-f5xk7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67265n\" (UID: \"9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67265n" Nov 02 22:52:58.656895 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.656862 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvtmn"] Nov 02 22:52:58.659392 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.659376 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvtmn" Nov 02 22:52:58.670321 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.670297 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67265n" Nov 02 22:52:58.674307 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.674282 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6wrp\" (UniqueName: \"kubernetes.io/projected/1389d802-5e2f-42df-b4b7-e10fccf8ceb7-kube-api-access-b6wrp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72f2rr\" (UID: \"1389d802-5e2f-42df-b4b7-e10fccf8ceb7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72f2rr" Nov 02 22:52:58.674414 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.674313 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q2l7\" (UniqueName: \"kubernetes.io/projected/6ba77a61-d033-4b7f-accf-2d842e49d934-kube-api-access-7q2l7\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv9n54\" (UID: \"6ba77a61-d033-4b7f-accf-2d842e49d934\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv9n54" Nov 02 22:52:58.674414 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.674344 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ba77a61-d033-4b7f-accf-2d842e49d934-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv9n54\" (UID: \"6ba77a61-d033-4b7f-accf-2d842e49d934\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv9n54" Nov 02 22:52:58.674526 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.674413 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1389d802-5e2f-42df-b4b7-e10fccf8ceb7-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72f2rr\" (UID: \"1389d802-5e2f-42df-b4b7-e10fccf8ceb7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72f2rr" Nov 02 22:52:58.674526 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.674456 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1389d802-5e2f-42df-b4b7-e10fccf8ceb7-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72f2rr\" (UID: \"1389d802-5e2f-42df-b4b7-e10fccf8ceb7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72f2rr" Nov 02 22:52:58.674526 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.674483 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ba77a61-d033-4b7f-accf-2d842e49d934-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv9n54\" (UID: \"6ba77a61-d033-4b7f-accf-2d842e49d934\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv9n54" Nov 02 22:52:58.674827 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.674809 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ba77a61-d033-4b7f-accf-2d842e49d934-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv9n54\" (UID: \"6ba77a61-d033-4b7f-accf-2d842e49d934\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv9n54" Nov 02 22:52:58.674882 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.674829 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ba77a61-d033-4b7f-accf-2d842e49d934-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv9n54\" (UID: \"6ba77a61-d033-4b7f-accf-2d842e49d934\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv9n54" Nov 02 22:52:58.713490 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.713457 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-7h5xq\"" Nov 02 22:52:58.720050 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.719954 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvtmn"] Nov 02 22:52:58.735880 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.735838 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q2l7\" (UniqueName: \"kubernetes.io/projected/6ba77a61-d033-4b7f-accf-2d842e49d934-kube-api-access-7q2l7\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv9n54\" (UID: \"6ba77a61-d033-4b7f-accf-2d842e49d934\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv9n54" Nov 02 22:52:58.769999 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.769967 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv9n54" Nov 02 22:52:58.775783 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.775761 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1389d802-5e2f-42df-b4b7-e10fccf8ceb7-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72f2rr\" (UID: \"1389d802-5e2f-42df-b4b7-e10fccf8ceb7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72f2rr" Nov 02 22:52:58.775855 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.775798 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj58t\" (UniqueName: \"kubernetes.io/projected/8559b43f-b9d0-4fed-90a0-c01922032b21-kube-api-access-bj58t\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvtmn\" (UID: \"8559b43f-b9d0-4fed-90a0-c01922032b21\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvtmn" Nov 02 22:52:58.775855 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.775837 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1389d802-5e2f-42df-b4b7-e10fccf8ceb7-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72f2rr\" (UID: \"1389d802-5e2f-42df-b4b7-e10fccf8ceb7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72f2rr" Nov 02 22:52:58.775978 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.775949 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8559b43f-b9d0-4fed-90a0-c01922032b21-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvtmn\" (UID: \"8559b43f-b9d0-4fed-90a0-c01922032b21\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvtmn" Nov 02 22:52:58.776059 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.776037 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8559b43f-b9d0-4fed-90a0-c01922032b21-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvtmn\" (UID: \"8559b43f-b9d0-4fed-90a0-c01922032b21\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvtmn" Nov 02 22:52:58.776108 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.776095 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6wrp\" (UniqueName: \"kubernetes.io/projected/1389d802-5e2f-42df-b4b7-e10fccf8ceb7-kube-api-access-b6wrp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72f2rr\" (UID: \"1389d802-5e2f-42df-b4b7-e10fccf8ceb7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72f2rr" Nov 02 22:52:58.776232 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.776196 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1389d802-5e2f-42df-b4b7-e10fccf8ceb7-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72f2rr\" (UID: \"1389d802-5e2f-42df-b4b7-e10fccf8ceb7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72f2rr" Nov 02 22:52:58.776232 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.776210 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1389d802-5e2f-42df-b4b7-e10fccf8ceb7-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72f2rr\" (UID: \"1389d802-5e2f-42df-b4b7-e10fccf8ceb7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72f2rr" Nov 02 22:52:58.829816 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.829788 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7ngb8"] Nov 02 22:52:58.833008 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:52:58.832978 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod495baab4_1e67_4202_8cb1_ab002c5dd19a.slice/crio-a781c56dedf1b1fe65ed2077b9951a71c7312df01f0c26618620f49f3c51303a WatchSource:0}: Error finding container a781c56dedf1b1fe65ed2077b9951a71c7312df01f0c26618620f49f3c51303a: Status 404 returned error can't find the container with id a781c56dedf1b1fe65ed2077b9951a71c7312df01f0c26618620f49f3c51303a Nov 02 22:52:58.837045 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.837022 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6wrp\" (UniqueName: \"kubernetes.io/projected/1389d802-5e2f-42df-b4b7-e10fccf8ceb7-kube-api-access-b6wrp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72f2rr\" (UID: \"1389d802-5e2f-42df-b4b7-e10fccf8ceb7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72f2rr" Nov 02 22:52:58.867739 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.867671 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72f2rr" Nov 02 22:52:58.876559 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.876529 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8559b43f-b9d0-4fed-90a0-c01922032b21-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvtmn\" (UID: \"8559b43f-b9d0-4fed-90a0-c01922032b21\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvtmn" Nov 02 22:52:58.876651 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.876563 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8559b43f-b9d0-4fed-90a0-c01922032b21-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvtmn\" (UID: \"8559b43f-b9d0-4fed-90a0-c01922032b21\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvtmn" Nov 02 22:52:58.876713 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.876629 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj58t\" (UniqueName: \"kubernetes.io/projected/8559b43f-b9d0-4fed-90a0-c01922032b21-kube-api-access-bj58t\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvtmn\" (UID: \"8559b43f-b9d0-4fed-90a0-c01922032b21\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvtmn" Nov 02 22:52:58.877046 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.876924 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8559b43f-b9d0-4fed-90a0-c01922032b21-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvtmn\" (UID: \"8559b43f-b9d0-4fed-90a0-c01922032b21\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvtmn" Nov 02 22:52:58.877046 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.876992 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8559b43f-b9d0-4fed-90a0-c01922032b21-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvtmn\" (UID: \"8559b43f-b9d0-4fed-90a0-c01922032b21\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvtmn" Nov 02 22:52:58.877731 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.877704 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7ngb8" event={"ID":"495baab4-1e67-4202-8cb1-ab002c5dd19a","Type":"ContainerStarted","Data":"a781c56dedf1b1fe65ed2077b9951a71c7312df01f0c26618620f49f3c51303a"} Nov 02 22:52:58.939929 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.939897 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj58t\" (UniqueName: \"kubernetes.io/projected/8559b43f-b9d0-4fed-90a0-c01922032b21-kube-api-access-bj58t\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvtmn\" (UID: \"8559b43f-b9d0-4fed-90a0-c01922032b21\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvtmn" Nov 02 22:52:58.972612 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:58.972580 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvtmn" Nov 02 22:52:59.043107 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:59.043070 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67265n"] Nov 02 22:52:59.046478 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:52:59.046450 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a110d03_4fe0_4d47_b9c0_fd1ca3ac180c.slice/crio-e2b32eccdbc2340d1439836282bdd96f753d956121733f3000043b7eaf53bef5 WatchSource:0}: Error finding container e2b32eccdbc2340d1439836282bdd96f753d956121733f3000043b7eaf53bef5: Status 404 returned error can't find the container with id e2b32eccdbc2340d1439836282bdd96f753d956121733f3000043b7eaf53bef5 Nov 02 22:52:59.139050 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:59.138975 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv9n54"] Nov 02 22:52:59.143408 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:52:59.143375 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba77a61_d033_4b7f_accf_2d842e49d934.slice/crio-d811225ea4455791d224a80f68ba12635696267c77c6dff0a0fd8d3ee86afddc WatchSource:0}: Error finding container d811225ea4455791d224a80f68ba12635696267c77c6dff0a0fd8d3ee86afddc: Status 404 returned error can't find the container with id d811225ea4455791d224a80f68ba12635696267c77c6dff0a0fd8d3ee86afddc Nov 02 22:52:59.180032 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:59.180005 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qgns6"] Nov 02 22:52:59.183159 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:59.183120 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qgns6" Nov 02 22:52:59.235264 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:59.235237 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-bztvn\"" Nov 02 22:52:59.236406 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:59.236385 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72f2rr"] Nov 02 22:52:59.243623 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:59.243586 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qgns6"] Nov 02 22:52:59.341559 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:59.341527 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvtmn"] Nov 02 22:52:59.344450 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:52:59.344420 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8559b43f_b9d0_4fed_90a0_c01922032b21.slice/crio-2353a2266bc999d5d20a586f73b1c8925c643eab5ee5040d7e4dc3122104c1f4 WatchSource:0}: Error finding container 2353a2266bc999d5d20a586f73b1c8925c643eab5ee5040d7e4dc3122104c1f4: Status 404 returned error can't find the container with id 2353a2266bc999d5d20a586f73b1c8925c643eab5ee5040d7e4dc3122104c1f4 Nov 02 22:52:59.380383 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:59.380356 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c416919-9bad-4354-a369-433901bc43a7-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qgns6\" (UID: \"0c416919-9bad-4354-a369-433901bc43a7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qgns6" Nov 02 22:52:59.380470 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:59.380386 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c416919-9bad-4354-a369-433901bc43a7-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qgns6\" (UID: \"0c416919-9bad-4354-a369-433901bc43a7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qgns6" Nov 02 22:52:59.380527 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:59.380478 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq64r\" (UniqueName: \"kubernetes.io/projected/0c416919-9bad-4354-a369-433901bc43a7-kube-api-access-lq64r\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qgns6\" (UID: \"0c416919-9bad-4354-a369-433901bc43a7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qgns6" Nov 02 22:52:59.481986 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:59.481908 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c416919-9bad-4354-a369-433901bc43a7-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qgns6\" (UID: \"0c416919-9bad-4354-a369-433901bc43a7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qgns6" Nov 02 22:52:59.481986 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:59.481948 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c416919-9bad-4354-a369-433901bc43a7-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qgns6\" (UID: \"0c416919-9bad-4354-a369-433901bc43a7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qgns6" Nov 02 22:52:59.482183 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:59.482011 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq64r\" (UniqueName: \"kubernetes.io/projected/0c416919-9bad-4354-a369-433901bc43a7-kube-api-access-lq64r\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qgns6\" (UID: \"0c416919-9bad-4354-a369-433901bc43a7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qgns6" Nov 02 22:52:59.482397 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:59.482376 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c416919-9bad-4354-a369-433901bc43a7-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qgns6\" (UID: \"0c416919-9bad-4354-a369-433901bc43a7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qgns6" Nov 02 22:52:59.482445 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:59.482386 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c416919-9bad-4354-a369-433901bc43a7-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qgns6\" (UID: \"0c416919-9bad-4354-a369-433901bc43a7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qgns6" Nov 02 22:52:59.543100 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:59.543072 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq64r\" (UniqueName: \"kubernetes.io/projected/0c416919-9bad-4354-a369-433901bc43a7-kube-api-access-lq64r\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qgns6\" (UID: \"0c416919-9bad-4354-a369-433901bc43a7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qgns6" Nov 02 22:52:59.796363 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:59.796275 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qgns6" Nov 02 22:52:59.887098 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:59.887046 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67265n" event={"ID":"9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c","Type":"ContainerStarted","Data":"e2b32eccdbc2340d1439836282bdd96f753d956121733f3000043b7eaf53bef5"} Nov 02 22:52:59.888417 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:59.888388 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvtmn" event={"ID":"8559b43f-b9d0-4fed-90a0-c01922032b21","Type":"ContainerStarted","Data":"2353a2266bc999d5d20a586f73b1c8925c643eab5ee5040d7e4dc3122104c1f4"} Nov 02 22:52:59.889795 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:59.889769 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72f2rr" event={"ID":"1389d802-5e2f-42df-b4b7-e10fccf8ceb7","Type":"ContainerStarted","Data":"7df35813b05f1af874a6c6bb74dcfef70f0e8a89596fd01cf0d953f27b828459"} Nov 02 22:52:59.890964 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:52:59.890939 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv9n54" event={"ID":"6ba77a61-d033-4b7f-accf-2d842e49d934","Type":"ContainerStarted","Data":"d811225ea4455791d224a80f68ba12635696267c77c6dff0a0fd8d3ee86afddc"} Nov 02 22:53:00.176839 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:00.176805 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qgns6"] Nov 02 22:53:00.181175 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:53:00.181122 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c416919_9bad_4354_a369_433901bc43a7.slice/crio-3b983f1c9a3f840d1a08704a9cabd0b67caa27242a643fe7b8a459eb54d74980 WatchSource:0}: Error finding container 3b983f1c9a3f840d1a08704a9cabd0b67caa27242a643fe7b8a459eb54d74980: Status 404 returned error can't find the container with id 3b983f1c9a3f840d1a08704a9cabd0b67caa27242a643fe7b8a459eb54d74980 Nov 02 22:53:00.899865 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:00.899825 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qgns6" event={"ID":"0c416919-9bad-4354-a369-433901bc43a7","Type":"ContainerStarted","Data":"3b983f1c9a3f840d1a08704a9cabd0b67caa27242a643fe7b8a459eb54d74980"} Nov 02 22:53:02.353114 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:02.351081 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c682r"] Nov 02 22:53:02.355037 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:02.354336 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c682r" Nov 02 22:53:02.413712 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:02.413462 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mp9t\" (UniqueName: \"kubernetes.io/projected/0ec8f93f-5c8f-4855-a174-96cad54fda07-kube-api-access-9mp9t\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c682r\" (UID: \"0ec8f93f-5c8f-4855-a174-96cad54fda07\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c682r" Nov 02 22:53:02.413712 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:02.413545 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ec8f93f-5c8f-4855-a174-96cad54fda07-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c682r\" (UID: \"0ec8f93f-5c8f-4855-a174-96cad54fda07\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c682r" Nov 02 22:53:02.413712 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:02.413572 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ec8f93f-5c8f-4855-a174-96cad54fda07-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c682r\" (UID: \"0ec8f93f-5c8f-4855-a174-96cad54fda07\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c682r" Nov 02 22:53:02.471998 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:02.470355 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-nf9z8\"" Nov 02 22:53:02.471998 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:02.471849 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c682r"] Nov 02 22:53:02.515049 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:02.515010 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ec8f93f-5c8f-4855-a174-96cad54fda07-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c682r\" (UID: \"0ec8f93f-5c8f-4855-a174-96cad54fda07\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c682r" Nov 02 22:53:02.515342 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:02.515060 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ec8f93f-5c8f-4855-a174-96cad54fda07-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c682r\" (UID: \"0ec8f93f-5c8f-4855-a174-96cad54fda07\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c682r" Nov 02 22:53:02.515342 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:02.515169 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mp9t\" (UniqueName: \"kubernetes.io/projected/0ec8f93f-5c8f-4855-a174-96cad54fda07-kube-api-access-9mp9t\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c682r\" (UID: \"0ec8f93f-5c8f-4855-a174-96cad54fda07\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c682r" Nov 02 22:53:02.515525 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:02.515490 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ec8f93f-5c8f-4855-a174-96cad54fda07-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c682r\" (UID: \"0ec8f93f-5c8f-4855-a174-96cad54fda07\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c682r" Nov 02 22:53:02.515793 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:02.515773 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ec8f93f-5c8f-4855-a174-96cad54fda07-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c682r\" (UID: \"0ec8f93f-5c8f-4855-a174-96cad54fda07\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c682r" Nov 02 22:53:02.589153 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:02.589031 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mp9t\" (UniqueName: \"kubernetes.io/projected/0ec8f93f-5c8f-4855-a174-96cad54fda07-kube-api-access-9mp9t\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c682r\" (UID: \"0ec8f93f-5c8f-4855-a174-96cad54fda07\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c682r" Nov 02 22:53:02.673760 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:02.673443 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c682r" Nov 02 22:53:03.165022 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:03.164987 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c682r"] Nov 02 22:53:03.173197 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:53:03.173163 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec8f93f_5c8f_4855_a174_96cad54fda07.slice/crio-42a29bca54932c3a793b99e07a299054e76335733b60da596c58d5cd9043ad26 WatchSource:0}: Error finding container 42a29bca54932c3a793b99e07a299054e76335733b60da596c58d5cd9043ad26: Status 404 returned error can't find the container with id 42a29bca54932c3a793b99e07a299054e76335733b60da596c58d5cd9043ad26 Nov 02 22:53:03.922447 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:03.922404 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c682r" event={"ID":"0ec8f93f-5c8f-4855-a174-96cad54fda07","Type":"ContainerStarted","Data":"42a29bca54932c3a793b99e07a299054e76335733b60da596c58d5cd9043ad26"} Nov 02 22:53:04.832890 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:04.831003 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bf4w9"] Nov 02 22:53:04.836639 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:04.836615 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bf4w9" Nov 02 22:53:04.861557 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:04.861510 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5nvcz"] Nov 02 22:53:04.865368 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:04.865348 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5nvcz" Nov 02 22:53:04.911737 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:04.911545 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bf4w9"] Nov 02 22:53:04.911737 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:04.911613 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn"] Nov 02 22:53:04.915326 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:04.914943 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn" Nov 02 22:53:04.946349 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:04.940447 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxhx76"] Nov 02 22:53:04.946349 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:04.945931 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n962\" (UniqueName: \"kubernetes.io/projected/d9ba2c71-9d0f-41f8-8f4c-b3b510183df3-kube-api-access-7n962\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn\" (UID: \"d9ba2c71-9d0f-41f8-8f4c-b3b510183df3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn" Nov 02 22:53:04.946349 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:04.945992 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9ba2c71-9d0f-41f8-8f4c-b3b510183df3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn\" (UID: \"d9ba2c71-9d0f-41f8-8f4c-b3b510183df3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn" Nov 02 22:53:04.946349 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:04.946033 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56404571-73cf-4705-89d1-3450f088978f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bf4w9\" (UID: \"56404571-73cf-4705-89d1-3450f088978f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bf4w9" Nov 02 22:53:04.946349 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:04.946097 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ts9w\" (UniqueName: \"kubernetes.io/projected/56404571-73cf-4705-89d1-3450f088978f-kube-api-access-7ts9w\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bf4w9\" (UID: \"56404571-73cf-4705-89d1-3450f088978f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bf4w9" Nov 02 22:53:04.946349 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:04.946154 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56404571-73cf-4705-89d1-3450f088978f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bf4w9\" (UID: \"56404571-73cf-4705-89d1-3450f088978f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bf4w9" Nov 02 22:53:04.946349 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:04.946173 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83f5166e-d9cc-45b1-9c5c-9189b8de96df-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5nvcz\" (UID: \"83f5166e-d9cc-45b1-9c5c-9189b8de96df\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5nvcz" Nov 02 22:53:04.946349 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:04.946199 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83f5166e-d9cc-45b1-9c5c-9189b8de96df-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5nvcz\" (UID: \"83f5166e-d9cc-45b1-9c5c-9189b8de96df\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5nvcz" Nov 02 22:53:04.946349 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:04.946267 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcwff\" (UniqueName: \"kubernetes.io/projected/83f5166e-d9cc-45b1-9c5c-9189b8de96df-kube-api-access-fcwff\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5nvcz\" (UID: \"83f5166e-d9cc-45b1-9c5c-9189b8de96df\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5nvcz" Nov 02 22:53:04.946349 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:04.946287 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9ba2c71-9d0f-41f8-8f4c-b3b510183df3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn\" (UID: \"d9ba2c71-9d0f-41f8-8f4c-b3b510183df3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn" Nov 02 22:53:04.964226 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:04.964175 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxhx76" Nov 02 22:53:05.007756 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.007719 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn"] Nov 02 22:53:05.009391 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.009368 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5nvcz"] Nov 02 22:53:05.033811 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.033159 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxhx76"] Nov 02 22:53:05.050181 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.047280 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83f5166e-d9cc-45b1-9c5c-9189b8de96df-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5nvcz\" (UID: \"83f5166e-d9cc-45b1-9c5c-9189b8de96df\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5nvcz" Nov 02 22:53:05.050181 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.047341 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcwff\" (UniqueName: \"kubernetes.io/projected/83f5166e-d9cc-45b1-9c5c-9189b8de96df-kube-api-access-fcwff\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5nvcz\" (UID: \"83f5166e-d9cc-45b1-9c5c-9189b8de96df\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5nvcz" Nov 02 22:53:05.050181 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.047373 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9ba2c71-9d0f-41f8-8f4c-b3b510183df3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn\" (UID: \"d9ba2c71-9d0f-41f8-8f4c-b3b510183df3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn" Nov 02 22:53:05.050181 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.047400 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26f14f8d-74a6-4c09-865b-b10bd97c4fb2-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxhx76\" (UID: \"26f14f8d-74a6-4c09-865b-b10bd97c4fb2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxhx76" Nov 02 22:53:05.050181 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.047437 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n962\" (UniqueName: \"kubernetes.io/projected/d9ba2c71-9d0f-41f8-8f4c-b3b510183df3-kube-api-access-7n962\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn\" (UID: \"d9ba2c71-9d0f-41f8-8f4c-b3b510183df3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn" Nov 02 22:53:05.050181 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.047481 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9ba2c71-9d0f-41f8-8f4c-b3b510183df3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn\" (UID: \"d9ba2c71-9d0f-41f8-8f4c-b3b510183df3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn" Nov 02 22:53:05.050181 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.047517 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56404571-73cf-4705-89d1-3450f088978f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bf4w9\" (UID: \"56404571-73cf-4705-89d1-3450f088978f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bf4w9" Nov 02 22:53:05.050181 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.047541 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ts9w\" (UniqueName: \"kubernetes.io/projected/56404571-73cf-4705-89d1-3450f088978f-kube-api-access-7ts9w\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bf4w9\" (UID: \"56404571-73cf-4705-89d1-3450f088978f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bf4w9" Nov 02 22:53:05.050181 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.047588 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26f14f8d-74a6-4c09-865b-b10bd97c4fb2-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxhx76\" (UID: \"26f14f8d-74a6-4c09-865b-b10bd97c4fb2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxhx76" Nov 02 22:53:05.050181 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.047633 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v62jh\" (UniqueName: \"kubernetes.io/projected/26f14f8d-74a6-4c09-865b-b10bd97c4fb2-kube-api-access-v62jh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxhx76\" (UID: \"26f14f8d-74a6-4c09-865b-b10bd97c4fb2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxhx76" Nov 02 22:53:05.050181 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.047662 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56404571-73cf-4705-89d1-3450f088978f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bf4w9\" (UID: \"56404571-73cf-4705-89d1-3450f088978f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bf4w9" Nov 02 22:53:05.050181 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.047689 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83f5166e-d9cc-45b1-9c5c-9189b8de96df-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5nvcz\" (UID: \"83f5166e-d9cc-45b1-9c5c-9189b8de96df\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5nvcz" Nov 02 22:53:05.050181 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.048101 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83f5166e-d9cc-45b1-9c5c-9189b8de96df-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5nvcz\" (UID: \"83f5166e-d9cc-45b1-9c5c-9189b8de96df\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5nvcz" Nov 02 22:53:05.050181 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.048383 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83f5166e-d9cc-45b1-9c5c-9189b8de96df-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5nvcz\" (UID: \"83f5166e-d9cc-45b1-9c5c-9189b8de96df\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5nvcz" Nov 02 22:53:05.050181 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.048932 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9ba2c71-9d0f-41f8-8f4c-b3b510183df3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn\" (UID: \"d9ba2c71-9d0f-41f8-8f4c-b3b510183df3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn" Nov 02 22:53:05.050181 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.049484 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9ba2c71-9d0f-41f8-8f4c-b3b510183df3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn\" (UID: \"d9ba2c71-9d0f-41f8-8f4c-b3b510183df3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn" Nov 02 22:53:05.050181 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.049786 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56404571-73cf-4705-89d1-3450f088978f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bf4w9\" (UID: \"56404571-73cf-4705-89d1-3450f088978f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bf4w9" Nov 02 22:53:05.051433 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.051346 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56404571-73cf-4705-89d1-3450f088978f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bf4w9\" (UID: \"56404571-73cf-4705-89d1-3450f088978f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bf4w9" Nov 02 22:53:05.119948 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.119821 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcwff\" (UniqueName: \"kubernetes.io/projected/83f5166e-d9cc-45b1-9c5c-9189b8de96df-kube-api-access-fcwff\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5nvcz\" (UID: \"83f5166e-d9cc-45b1-9c5c-9189b8de96df\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5nvcz" Nov 02 22:53:05.121636 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.121612 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n962\" (UniqueName: \"kubernetes.io/projected/d9ba2c71-9d0f-41f8-8f4c-b3b510183df3-kube-api-access-7n962\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn\" (UID: \"d9ba2c71-9d0f-41f8-8f4c-b3b510183df3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn" Nov 02 22:53:05.127976 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.127951 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ts9w\" (UniqueName: \"kubernetes.io/projected/56404571-73cf-4705-89d1-3450f088978f-kube-api-access-7ts9w\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bf4w9\" (UID: \"56404571-73cf-4705-89d1-3450f088978f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bf4w9" Nov 02 22:53:05.148275 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.148246 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26f14f8d-74a6-4c09-865b-b10bd97c4fb2-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxhx76\" (UID: \"26f14f8d-74a6-4c09-865b-b10bd97c4fb2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxhx76" Nov 02 22:53:05.148376 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.148331 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26f14f8d-74a6-4c09-865b-b10bd97c4fb2-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxhx76\" (UID: \"26f14f8d-74a6-4c09-865b-b10bd97c4fb2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxhx76" Nov 02 22:53:05.148444 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.148373 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v62jh\" (UniqueName: \"kubernetes.io/projected/26f14f8d-74a6-4c09-865b-b10bd97c4fb2-kube-api-access-v62jh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxhx76\" (UID: \"26f14f8d-74a6-4c09-865b-b10bd97c4fb2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxhx76" Nov 02 22:53:05.149091 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.149065 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26f14f8d-74a6-4c09-865b-b10bd97c4fb2-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxhx76\" (UID: \"26f14f8d-74a6-4c09-865b-b10bd97c4fb2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxhx76" Nov 02 22:53:05.149212 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.149110 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26f14f8d-74a6-4c09-865b-b10bd97c4fb2-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxhx76\" (UID: \"26f14f8d-74a6-4c09-865b-b10bd97c4fb2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxhx76" Nov 02 22:53:05.156238 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.156203 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bf4w9" Nov 02 22:53:05.185698 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.185521 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5nvcz" Nov 02 22:53:05.216904 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.216863 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v62jh\" (UniqueName: \"kubernetes.io/projected/26f14f8d-74a6-4c09-865b-b10bd97c4fb2-kube-api-access-v62jh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxhx76\" (UID: \"26f14f8d-74a6-4c09-865b-b10bd97c4fb2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxhx76" Nov 02 22:53:05.242464 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.242241 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn" Nov 02 22:53:05.289206 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.288929 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxhx76" Nov 02 22:53:05.476771 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.476739 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh"] Nov 02 22:53:05.482277 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.481748 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:05.535891 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.535697 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 22:53:05.536368 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.536292 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-98x6w\"" Nov 02 22:53:05.542191 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.542168 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh"] Nov 02 22:53:05.553527 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.553487 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4c2f4f5-743d-4e9d-ae63-1769c521b4ad-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-gcxrh\" (UID: \"c4c2f4f5-743d-4e9d-ae63-1769c521b4ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:05.553661 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.553533 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4c2f4f5-743d-4e9d-ae63-1769c521b4ad-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-gcxrh\" (UID: \"c4c2f4f5-743d-4e9d-ae63-1769c521b4ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:05.553661 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.553567 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mdjf\" (UniqueName: \"kubernetes.io/projected/c4c2f4f5-743d-4e9d-ae63-1769c521b4ad-kube-api-access-8mdjf\") pod \"maas-default-gateway-openshift-default-694db55c87-gcxrh\" (UID: \"c4c2f4f5-743d-4e9d-ae63-1769c521b4ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:05.553661 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.553635 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4c2f4f5-743d-4e9d-ae63-1769c521b4ad-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-gcxrh\" (UID: \"c4c2f4f5-743d-4e9d-ae63-1769c521b4ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:05.553817 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.553666 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4c2f4f5-743d-4e9d-ae63-1769c521b4ad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-gcxrh\" (UID: \"c4c2f4f5-743d-4e9d-ae63-1769c521b4ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:05.553817 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.553712 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4c2f4f5-743d-4e9d-ae63-1769c521b4ad-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-gcxrh\" (UID: \"c4c2f4f5-743d-4e9d-ae63-1769c521b4ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:05.553817 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.553743 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4c2f4f5-743d-4e9d-ae63-1769c521b4ad-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-gcxrh\" (UID: \"c4c2f4f5-743d-4e9d-ae63-1769c521b4ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:05.553817 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.553790 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4c2f4f5-743d-4e9d-ae63-1769c521b4ad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-gcxrh\" (UID: \"c4c2f4f5-743d-4e9d-ae63-1769c521b4ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:05.553999 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.553816 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4c2f4f5-743d-4e9d-ae63-1769c521b4ad-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-gcxrh\" (UID: \"c4c2f4f5-743d-4e9d-ae63-1769c521b4ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:05.589100 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.588163 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bf4w9"] Nov 02 22:53:05.593748 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:53:05.593716 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56404571_73cf_4705_89d1_3450f088978f.slice/crio-8bb45241d25967fb8b4d3b51b273de17da398f4a9cacc3479fb708680f633891 WatchSource:0}: Error finding container 8bb45241d25967fb8b4d3b51b273de17da398f4a9cacc3479fb708680f633891: Status 404 returned error can't find the container with id 8bb45241d25967fb8b4d3b51b273de17da398f4a9cacc3479fb708680f633891 Nov 02 22:53:05.650452 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.650351 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5nvcz"] Nov 02 22:53:05.654551 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.654521 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4c2f4f5-743d-4e9d-ae63-1769c521b4ad-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-gcxrh\" (UID: \"c4c2f4f5-743d-4e9d-ae63-1769c521b4ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:05.654697 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.654577 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4c2f4f5-743d-4e9d-ae63-1769c521b4ad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-gcxrh\" (UID: \"c4c2f4f5-743d-4e9d-ae63-1769c521b4ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:05.654697 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.654608 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4c2f4f5-743d-4e9d-ae63-1769c521b4ad-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-gcxrh\" (UID: \"c4c2f4f5-743d-4e9d-ae63-1769c521b4ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:05.654697 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.654645 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4c2f4f5-743d-4e9d-ae63-1769c521b4ad-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-gcxrh\" (UID: \"c4c2f4f5-743d-4e9d-ae63-1769c521b4ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:05.654697 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.654676 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4c2f4f5-743d-4e9d-ae63-1769c521b4ad-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-gcxrh\" (UID: \"c4c2f4f5-743d-4e9d-ae63-1769c521b4ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:05.654921 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.654705 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mdjf\" (UniqueName: \"kubernetes.io/projected/c4c2f4f5-743d-4e9d-ae63-1769c521b4ad-kube-api-access-8mdjf\") pod \"maas-default-gateway-openshift-default-694db55c87-gcxrh\" (UID: \"c4c2f4f5-743d-4e9d-ae63-1769c521b4ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:05.654921 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.654767 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4c2f4f5-743d-4e9d-ae63-1769c521b4ad-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-gcxrh\" (UID: \"c4c2f4f5-743d-4e9d-ae63-1769c521b4ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:05.654921 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.654794 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4c2f4f5-743d-4e9d-ae63-1769c521b4ad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-gcxrh\" (UID: \"c4c2f4f5-743d-4e9d-ae63-1769c521b4ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:05.654921 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.654839 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4c2f4f5-743d-4e9d-ae63-1769c521b4ad-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-gcxrh\" (UID: \"c4c2f4f5-743d-4e9d-ae63-1769c521b4ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:05.655265 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.655244 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4c2f4f5-743d-4e9d-ae63-1769c521b4ad-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-gcxrh\" (UID: \"c4c2f4f5-743d-4e9d-ae63-1769c521b4ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:05.655514 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.655494 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4c2f4f5-743d-4e9d-ae63-1769c521b4ad-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-gcxrh\" (UID: \"c4c2f4f5-743d-4e9d-ae63-1769c521b4ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:05.656152 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.656107 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4c2f4f5-743d-4e9d-ae63-1769c521b4ad-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-gcxrh\" (UID: \"c4c2f4f5-743d-4e9d-ae63-1769c521b4ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:05.656654 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.656632 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4c2f4f5-743d-4e9d-ae63-1769c521b4ad-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-gcxrh\" (UID: \"c4c2f4f5-743d-4e9d-ae63-1769c521b4ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:05.657579 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.657556 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4c2f4f5-743d-4e9d-ae63-1769c521b4ad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-gcxrh\" (UID: \"c4c2f4f5-743d-4e9d-ae63-1769c521b4ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:05.659779 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.659756 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4c2f4f5-743d-4e9d-ae63-1769c521b4ad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-gcxrh\" (UID: \"c4c2f4f5-743d-4e9d-ae63-1769c521b4ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:05.659954 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.659929 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4c2f4f5-743d-4e9d-ae63-1769c521b4ad-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-gcxrh\" (UID: \"c4c2f4f5-743d-4e9d-ae63-1769c521b4ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:05.674251 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:53:05.674199 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f5166e_d9cc_45b1_9c5c_9189b8de96df.slice/crio-ff412771ba5ce1abcb4e0209a106d7a7e9d7d4d1816004ab177d0f34bb8d2263 WatchSource:0}: Error finding container ff412771ba5ce1abcb4e0209a106d7a7e9d7d4d1816004ab177d0f34bb8d2263: Status 404 returned error can't find the container with id ff412771ba5ce1abcb4e0209a106d7a7e9d7d4d1816004ab177d0f34bb8d2263 Nov 02 22:53:05.690409 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.690378 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn"] Nov 02 22:53:05.698549 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:53:05.698520 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9ba2c71_9d0f_41f8_8f4c_b3b510183df3.slice/crio-dd7bca809dd188377c6baf4e004ed71c64f7181497ff2dcb567686e1755d2f9f WatchSource:0}: Error finding container dd7bca809dd188377c6baf4e004ed71c64f7181497ff2dcb567686e1755d2f9f: Status 404 returned error can't find the container with id dd7bca809dd188377c6baf4e004ed71c64f7181497ff2dcb567686e1755d2f9f Nov 02 22:53:05.710017 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.709945 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxhx76"] Nov 02 22:53:05.721777 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:53:05.721263 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f14f8d_74a6_4c09_865b_b10bd97c4fb2.slice/crio-f62c89e158f132812efe1eb7279adf08e2fbe96bf40fad05214d7a9bc7b5293c WatchSource:0}: Error finding container f62c89e158f132812efe1eb7279adf08e2fbe96bf40fad05214d7a9bc7b5293c: Status 404 returned error can't find the container with id f62c89e158f132812efe1eb7279adf08e2fbe96bf40fad05214d7a9bc7b5293c Nov 02 22:53:05.721777 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.721714 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mdjf\" (UniqueName: \"kubernetes.io/projected/c4c2f4f5-743d-4e9d-ae63-1769c521b4ad-kube-api-access-8mdjf\") pod \"maas-default-gateway-openshift-default-694db55c87-gcxrh\" (UID: \"c4c2f4f5-743d-4e9d-ae63-1769c521b4ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:05.727210 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.727183 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4c2f4f5-743d-4e9d-ae63-1769c521b4ad-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-gcxrh\" (UID: \"c4c2f4f5-743d-4e9d-ae63-1769c521b4ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:05.793225 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.793158 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2"] Nov 02 22:53:05.802091 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.802058 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:05.802685 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.802663 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:05.855956 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.855920 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f85494-1949-4969-8e31-c6a4b155f4ae-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2\" (UID: \"59f85494-1949-4969-8e31-c6a4b155f4ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:05.856114 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.855966 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f85494-1949-4969-8e31-c6a4b155f4ae-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2\" (UID: \"59f85494-1949-4969-8e31-c6a4b155f4ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:05.856114 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.856042 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f85494-1949-4969-8e31-c6a4b155f4ae-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2\" (UID: \"59f85494-1949-4969-8e31-c6a4b155f4ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:05.856114 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.856070 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f85494-1949-4969-8e31-c6a4b155f4ae-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2\" (UID: \"59f85494-1949-4969-8e31-c6a4b155f4ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:05.856114 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.856096 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f85494-1949-4969-8e31-c6a4b155f4ae-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2\" (UID: \"59f85494-1949-4969-8e31-c6a4b155f4ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:05.856114 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.856120 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f85494-1949-4969-8e31-c6a4b155f4ae-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2\" (UID: \"59f85494-1949-4969-8e31-c6a4b155f4ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:05.856114 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.856191 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sx7g\" (UniqueName: \"kubernetes.io/projected/59f85494-1949-4969-8e31-c6a4b155f4ae-kube-api-access-9sx7g\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2\" (UID: \"59f85494-1949-4969-8e31-c6a4b155f4ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:05.856655 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.856251 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f85494-1949-4969-8e31-c6a4b155f4ae-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2\" (UID: \"59f85494-1949-4969-8e31-c6a4b155f4ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:05.856655 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.856277 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f85494-1949-4969-8e31-c6a4b155f4ae-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2\" (UID: \"59f85494-1949-4969-8e31-c6a4b155f4ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:05.859938 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.859828 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-htjf2\"" Nov 02 22:53:05.872019 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.871939 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2"] Nov 02 22:53:05.948582 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.946853 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5nvcz" event={"ID":"83f5166e-d9cc-45b1-9c5c-9189b8de96df","Type":"ContainerStarted","Data":"ff412771ba5ce1abcb4e0209a106d7a7e9d7d4d1816004ab177d0f34bb8d2263"} Nov 02 22:53:05.949292 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.949092 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn" event={"ID":"d9ba2c71-9d0f-41f8-8f4c-b3b510183df3","Type":"ContainerStarted","Data":"dd7bca809dd188377c6baf4e004ed71c64f7181497ff2dcb567686e1755d2f9f"} Nov 02 22:53:05.952924 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.952896 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxhx76" event={"ID":"26f14f8d-74a6-4c09-865b-b10bd97c4fb2","Type":"ContainerStarted","Data":"f62c89e158f132812efe1eb7279adf08e2fbe96bf40fad05214d7a9bc7b5293c"} Nov 02 22:53:05.957782 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.957726 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bf4w9" event={"ID":"56404571-73cf-4705-89d1-3450f088978f","Type":"ContainerStarted","Data":"8bb45241d25967fb8b4d3b51b273de17da398f4a9cacc3479fb708680f633891"} Nov 02 22:53:05.961188 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.958341 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f85494-1949-4969-8e31-c6a4b155f4ae-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2\" (UID: \"59f85494-1949-4969-8e31-c6a4b155f4ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:05.961188 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.958379 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f85494-1949-4969-8e31-c6a4b155f4ae-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2\" (UID: \"59f85494-1949-4969-8e31-c6a4b155f4ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:05.961188 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.958410 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f85494-1949-4969-8e31-c6a4b155f4ae-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2\" (UID: \"59f85494-1949-4969-8e31-c6a4b155f4ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:05.961188 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.958436 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f85494-1949-4969-8e31-c6a4b155f4ae-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2\" (UID: \"59f85494-1949-4969-8e31-c6a4b155f4ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:05.961188 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.958462 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sx7g\" (UniqueName: \"kubernetes.io/projected/59f85494-1949-4969-8e31-c6a4b155f4ae-kube-api-access-9sx7g\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2\" (UID: \"59f85494-1949-4969-8e31-c6a4b155f4ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:05.961188 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.958517 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f85494-1949-4969-8e31-c6a4b155f4ae-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2\" (UID: \"59f85494-1949-4969-8e31-c6a4b155f4ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:05.961188 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.958544 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f85494-1949-4969-8e31-c6a4b155f4ae-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2\" (UID: \"59f85494-1949-4969-8e31-c6a4b155f4ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:05.961188 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.958608 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f85494-1949-4969-8e31-c6a4b155f4ae-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2\" (UID: \"59f85494-1949-4969-8e31-c6a4b155f4ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:05.961188 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.958636 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f85494-1949-4969-8e31-c6a4b155f4ae-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2\" (UID: \"59f85494-1949-4969-8e31-c6a4b155f4ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:05.961188 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.959178 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f85494-1949-4969-8e31-c6a4b155f4ae-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2\" (UID: \"59f85494-1949-4969-8e31-c6a4b155f4ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:05.963589 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.962464 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f85494-1949-4969-8e31-c6a4b155f4ae-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2\" (UID: \"59f85494-1949-4969-8e31-c6a4b155f4ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:05.963589 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.963008 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f85494-1949-4969-8e31-c6a4b155f4ae-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2\" (UID: \"59f85494-1949-4969-8e31-c6a4b155f4ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:05.963739 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.963672 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f85494-1949-4969-8e31-c6a4b155f4ae-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2\" (UID: \"59f85494-1949-4969-8e31-c6a4b155f4ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:05.966710 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.966680 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f85494-1949-4969-8e31-c6a4b155f4ae-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2\" (UID: \"59f85494-1949-4969-8e31-c6a4b155f4ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:05.973742 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.973716 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f85494-1949-4969-8e31-c6a4b155f4ae-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2\" (UID: \"59f85494-1949-4969-8e31-c6a4b155f4ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:05.975946 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:05.975765 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f85494-1949-4969-8e31-c6a4b155f4ae-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2\" (UID: \"59f85494-1949-4969-8e31-c6a4b155f4ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:06.025474 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:06.025376 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sx7g\" (UniqueName: \"kubernetes.io/projected/59f85494-1949-4969-8e31-c6a4b155f4ae-kube-api-access-9sx7g\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2\" (UID: \"59f85494-1949-4969-8e31-c6a4b155f4ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:06.031624 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:06.031531 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f85494-1949-4969-8e31-c6a4b155f4ae-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2\" (UID: \"59f85494-1949-4969-8e31-c6a4b155f4ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:06.122198 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:06.121951 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:06.273455 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:06.273366 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh"] Nov 02 22:53:06.284419 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:53:06.284379 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4c2f4f5_743d_4e9d_ae63_1769c521b4ad.slice/crio-5662149e8311fe61181b09bbdb10b0344a3a0e81c7b6cd16519ba4642b060d3e WatchSource:0}: Error finding container 5662149e8311fe61181b09bbdb10b0344a3a0e81c7b6cd16519ba4642b060d3e: Status 404 returned error can't find the container with id 5662149e8311fe61181b09bbdb10b0344a3a0e81c7b6cd16519ba4642b060d3e Nov 02 22:53:06.554810 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:06.554701 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2"] Nov 02 22:53:06.575803 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:53:06.575767 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f85494_1949_4969_8e31_c6a4b155f4ae.slice/crio-c8337e4dd3cc6029039e19d09828bab016a0e24092be27e68568c31731be5b3d WatchSource:0}: Error finding container c8337e4dd3cc6029039e19d09828bab016a0e24092be27e68568c31731be5b3d: Status 404 returned error can't find the container with id c8337e4dd3cc6029039e19d09828bab016a0e24092be27e68568c31731be5b3d Nov 02 22:53:06.978869 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:06.978790 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" event={"ID":"c4c2f4f5-743d-4e9d-ae63-1769c521b4ad","Type":"ContainerStarted","Data":"5662149e8311fe61181b09bbdb10b0344a3a0e81c7b6cd16519ba4642b060d3e"} Nov 02 22:53:06.985846 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:06.985812 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" event={"ID":"59f85494-1949-4969-8e31-c6a4b155f4ae","Type":"ContainerStarted","Data":"c8337e4dd3cc6029039e19d09828bab016a0e24092be27e68568c31731be5b3d"} Nov 02 22:53:20.086995 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:20.086569 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 22:53:20.086995 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:20.086646 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 22:53:20.086995 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:20.086684 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 22:53:20.099584 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:20.099552 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 22:53:20.099710 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:20.099628 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 22:53:20.099710 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:20.099669 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 22:53:20.200566 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:20.200535 5727 generic.go:356] "Generic (PLEG): container finished" podID="495baab4-1e67-4202-8cb1-ab002c5dd19a" containerID="28e0c89292c874ad059dadb9124cdee5787641ce7ca5c3736f1125eae580a0a7" exitCode=0 Nov 02 22:53:20.200711 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:20.200666 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7ngb8" event={"ID":"495baab4-1e67-4202-8cb1-ab002c5dd19a","Type":"ContainerDied","Data":"28e0c89292c874ad059dadb9124cdee5787641ce7ca5c3736f1125eae580a0a7"} Nov 02 22:53:20.204872 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:20.204727 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" event={"ID":"c4c2f4f5-743d-4e9d-ae63-1769c521b4ad","Type":"ContainerStarted","Data":"b0541cd249a21e0f7d46bdf6006d64ab830a350f3213ea0df81c1a2234d21d47"} Nov 02 22:53:20.207048 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:20.207021 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" event={"ID":"59f85494-1949-4969-8e31-c6a4b155f4ae","Type":"ContainerStarted","Data":"124d60f4f4da20b2cb1e10838a16fb265e4339a7d959c4b06bac0e06f7813484"} Nov 02 22:53:20.490667 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:20.490561 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" podStartSLOduration=1.691989839 podStartE2EDuration="15.490542254s" totalImagesPullingTime="13.798552415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:53:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:20.486657 +0000 UTC m=+286.725154677" watchObservedRunningTime="2026-11-02 22:53:20.490542254 +0000 UTC m=+286.729039916" Nov 02 22:53:20.607580 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:20.607528 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" podStartSLOduration=2.097932091 podStartE2EDuration="15.607510746s" totalImagesPullingTime="13.509578655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:53:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:20.605737026 +0000 UTC m=+286.844234713" watchObservedRunningTime="2026-11-02 22:53:20.607510746 +0000 UTC m=+286.846008409" Nov 02 22:53:20.803142 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:20.803108 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:20.803250 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:20.803195 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:21.122487 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:21.122452 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:21.122866 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:21.122563 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:21.128944 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:21.128912 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:21.213425 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:21.213394 5727 generic.go:356] "Generic (PLEG): container finished" podID="1389d802-5e2f-42df-b4b7-e10fccf8ceb7" containerID="a5f413c887752c09c6c1f1b48f0091566fc047f8d055a7c4683a5250886da634" exitCode=0 Nov 02 22:53:21.213555 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:21.213500 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72f2rr" event={"ID":"1389d802-5e2f-42df-b4b7-e10fccf8ceb7","Type":"ContainerDied","Data":"a5f413c887752c09c6c1f1b48f0091566fc047f8d055a7c4683a5250886da634"} Nov 02 22:53:21.215085 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:21.215064 5727 generic.go:356] "Generic (PLEG): container finished" podID="6ba77a61-d033-4b7f-accf-2d842e49d934" containerID="7401b90bf070d68d72f0ebeb9ba832ae8d07d76fe2d23a7a1d59f52120318ce0" exitCode=0 Nov 02 22:53:21.215188 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:21.215094 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv9n54" event={"ID":"6ba77a61-d033-4b7f-accf-2d842e49d934","Type":"ContainerDied","Data":"7401b90bf070d68d72f0ebeb9ba832ae8d07d76fe2d23a7a1d59f52120318ce0"} Nov 02 22:53:21.217435 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:21.217415 5727 generic.go:356] "Generic (PLEG): container finished" podID="9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c" containerID="a43d18133936232d518eb15b4a0b078cb9e2c7e29de09924ae205781df6ac64d" exitCode=0 Nov 02 22:53:21.217574 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:21.217507 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67265n" event={"ID":"9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c","Type":"ContainerDied","Data":"a43d18133936232d518eb15b4a0b078cb9e2c7e29de09924ae205781df6ac64d"} Nov 02 22:53:21.219297 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:21.219260 5727 generic.go:356] "Generic (PLEG): container finished" podID="26f14f8d-74a6-4c09-865b-b10bd97c4fb2" containerID="7ef89f4310a87dd0fa9faf1bad092e75f4c24d55b13649a589d9575a67a91d84" exitCode=0 Nov 02 22:53:21.219443 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:21.219330 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxhx76" event={"ID":"26f14f8d-74a6-4c09-865b-b10bd97c4fb2","Type":"ContainerDied","Data":"7ef89f4310a87dd0fa9faf1bad092e75f4c24d55b13649a589d9575a67a91d84"} Nov 02 22:53:21.221542 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:21.221520 5727 generic.go:356] "Generic (PLEG): container finished" podID="56404571-73cf-4705-89d1-3450f088978f" containerID="986488d22c54bddf3b481b69c6de35725d96f026fe6cf8a112e16c34a2967ba7" exitCode=0 Nov 02 22:53:21.221649 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:21.221598 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bf4w9" event={"ID":"56404571-73cf-4705-89d1-3450f088978f","Type":"ContainerDied","Data":"986488d22c54bddf3b481b69c6de35725d96f026fe6cf8a112e16c34a2967ba7"} Nov 02 22:53:21.224178 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:21.223938 5727 generic.go:356] "Generic (PLEG): container finished" podID="0c416919-9bad-4354-a369-433901bc43a7" containerID="6546fe8b6e4e99187cc08dd72d8173621420a339c7996e4723a0b45096b07043" exitCode=0 Nov 02 22:53:21.224178 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:21.224013 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qgns6" event={"ID":"0c416919-9bad-4354-a369-433901bc43a7","Type":"ContainerDied","Data":"6546fe8b6e4e99187cc08dd72d8173621420a339c7996e4723a0b45096b07043"} Nov 02 22:53:21.225620 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:21.225598 5727 generic.go:356] "Generic (PLEG): container finished" podID="83f5166e-d9cc-45b1-9c5c-9189b8de96df" containerID="2339f691c5533008bfb3df955912e30dbab8ee9bf146214b296656e805b896f6" exitCode=0 Nov 02 22:53:21.225699 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:21.225667 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5nvcz" event={"ID":"83f5166e-d9cc-45b1-9c5c-9189b8de96df","Type":"ContainerDied","Data":"2339f691c5533008bfb3df955912e30dbab8ee9bf146214b296656e805b896f6"} Nov 02 22:53:21.228028 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:21.228015 5727 generic.go:356] "Generic (PLEG): container finished" podID="8559b43f-b9d0-4fed-90a0-c01922032b21" containerID="6cb81f2be2f0df7fbeddb962ac3ab3949a59252781016b028f74a6b76f9ba00d" exitCode=0 Nov 02 22:53:21.228170 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:21.228111 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvtmn" event={"ID":"8559b43f-b9d0-4fed-90a0-c01922032b21","Type":"ContainerDied","Data":"6cb81f2be2f0df7fbeddb962ac3ab3949a59252781016b028f74a6b76f9ba00d"} Nov 02 22:53:21.229506 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:21.229490 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2" Nov 02 22:53:21.809212 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:21.809183 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:22.234924 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:22.234837 5727 generic.go:356] "Generic (PLEG): container finished" podID="0ec8f93f-5c8f-4855-a174-96cad54fda07" containerID="cdddbdeacb28031c734c688c66d5d0385e0f716caef2d744ab82d00fd62cd993" exitCode=0 Nov 02 22:53:22.235350 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:22.234919 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c682r" event={"ID":"0ec8f93f-5c8f-4855-a174-96cad54fda07","Type":"ContainerDied","Data":"cdddbdeacb28031c734c688c66d5d0385e0f716caef2d744ab82d00fd62cd993"} Nov 02 22:53:22.236728 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:22.236652 5727 generic.go:356] "Generic (PLEG): container finished" podID="d9ba2c71-9d0f-41f8-8f4c-b3b510183df3" containerID="7df5268408533ab76c0ca4aa6529e19ab3c60cb0ca39f550b5e3b2fa27cf1d69" exitCode=0 Nov 02 22:53:22.236791 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:22.236747 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn" event={"ID":"d9ba2c71-9d0f-41f8-8f4c-b3b510183df3","Type":"ContainerDied","Data":"7df5268408533ab76c0ca4aa6529e19ab3c60cb0ca39f550b5e3b2fa27cf1d69"} Nov 02 22:53:22.238092 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:22.238074 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:53:23.242395 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:23.242366 5727 generic.go:356] "Generic (PLEG): container finished" podID="0ec8f93f-5c8f-4855-a174-96cad54fda07" containerID="4918cefeb3840a1485addb5246cac547b8435b4d0303ef8a11daeac8e4bae4b6" exitCode=0 Nov 02 22:53:23.242741 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:23.242453 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c682r" event={"ID":"0ec8f93f-5c8f-4855-a174-96cad54fda07","Type":"ContainerDied","Data":"4918cefeb3840a1485addb5246cac547b8435b4d0303ef8a11daeac8e4bae4b6"} Nov 02 22:53:24.254838 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:24.254803 5727 generic.go:356] "Generic (PLEG): container finished" podID="1389d802-5e2f-42df-b4b7-e10fccf8ceb7" containerID="534a0eb6a4e4ef9aed191b4c0e8c0985fc5155ef53ee715e5d3fe97ef0cc8fde" exitCode=0 Nov 02 22:53:24.255583 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:24.254932 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72f2rr" event={"ID":"1389d802-5e2f-42df-b4b7-e10fccf8ceb7","Type":"ContainerDied","Data":"534a0eb6a4e4ef9aed191b4c0e8c0985fc5155ef53ee715e5d3fe97ef0cc8fde"} Nov 02 22:53:24.261466 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:24.260362 5727 generic.go:356] "Generic (PLEG): container finished" podID="56404571-73cf-4705-89d1-3450f088978f" containerID="3262903d90e0c5d3aeeab259cb3a2eda37c4d4662d7aabf261ccdd7dd853b342" exitCode=0 Nov 02 22:53:24.261466 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:24.261413 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bf4w9" event={"ID":"56404571-73cf-4705-89d1-3450f088978f","Type":"ContainerDied","Data":"3262903d90e0c5d3aeeab259cb3a2eda37c4d4662d7aabf261ccdd7dd853b342"} Nov 02 22:53:25.268999 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:25.268957 5727 generic.go:356] "Generic (PLEG): container finished" podID="6ba77a61-d033-4b7f-accf-2d842e49d934" containerID="b50a0fa0ec98d25ad080ce6a7abf4be8461e16cd9d32bb2aba319469b0fd1f49" exitCode=0 Nov 02 22:53:25.269497 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:25.269052 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv9n54" event={"ID":"6ba77a61-d033-4b7f-accf-2d842e49d934","Type":"ContainerDied","Data":"b50a0fa0ec98d25ad080ce6a7abf4be8461e16cd9d32bb2aba319469b0fd1f49"} Nov 02 22:53:25.275637 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:25.275602 5727 generic.go:356] "Generic (PLEG): container finished" podID="d9ba2c71-9d0f-41f8-8f4c-b3b510183df3" containerID="fe60ed6cb978273ee591f1f6aa25ab1f5ef59cc122a0217fd2a6f6e9729820e6" exitCode=0 Nov 02 22:53:25.276367 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:25.275790 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn" event={"ID":"d9ba2c71-9d0f-41f8-8f4c-b3b510183df3","Type":"ContainerDied","Data":"fe60ed6cb978273ee591f1f6aa25ab1f5ef59cc122a0217fd2a6f6e9729820e6"} Nov 02 22:53:25.279082 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:25.279058 5727 generic.go:356] "Generic (PLEG): container finished" podID="9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c" containerID="5e9e67c5f8e8b412237f79bbe7ac33ce6ba436d50776649587c9b940a018d47d" exitCode=0 Nov 02 22:53:25.279214 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:25.279167 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67265n" event={"ID":"9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c","Type":"ContainerDied","Data":"5e9e67c5f8e8b412237f79bbe7ac33ce6ba436d50776649587c9b940a018d47d"} Nov 02 22:53:25.282821 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:25.282644 5727 generic.go:356] "Generic (PLEG): container finished" podID="26f14f8d-74a6-4c09-865b-b10bd97c4fb2" containerID="6f8cd1d464da2371e4825b1ad81a02f6ada6889ac4131fc433d6da2e380a5446" exitCode=0 Nov 02 22:53:25.282821 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:25.282744 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxhx76" event={"ID":"26f14f8d-74a6-4c09-865b-b10bd97c4fb2","Type":"ContainerDied","Data":"6f8cd1d464da2371e4825b1ad81a02f6ada6889ac4131fc433d6da2e380a5446"} Nov 02 22:53:25.286766 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:25.286726 5727 generic.go:356] "Generic (PLEG): container finished" podID="83f5166e-d9cc-45b1-9c5c-9189b8de96df" containerID="6ede49acb2448c63ac6a26981f9fd7b1f4f7cc857baf6dbca5fea5f1e9c25440" exitCode=0 Nov 02 22:53:25.286841 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:25.286799 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5nvcz" event={"ID":"83f5166e-d9cc-45b1-9c5c-9189b8de96df","Type":"ContainerDied","Data":"6ede49acb2448c63ac6a26981f9fd7b1f4f7cc857baf6dbca5fea5f1e9c25440"} Nov 02 22:53:26.301332 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:26.300470 5727 generic.go:356] "Generic (PLEG): container finished" podID="8559b43f-b9d0-4fed-90a0-c01922032b21" containerID="cbb1f262f6a2270ba53084d548bc26f43915fcf0a8b8f80ad0b1f2d5fc9f8ad1" exitCode=0 Nov 02 22:53:26.301332 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:26.300555 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvtmn" event={"ID":"8559b43f-b9d0-4fed-90a0-c01922032b21","Type":"ContainerDied","Data":"cbb1f262f6a2270ba53084d548bc26f43915fcf0a8b8f80ad0b1f2d5fc9f8ad1"} Nov 02 22:53:27.867023 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:53:27.866939 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 22:53:29.353621 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:29.353573 5727 generic.go:356] "Generic (PLEG): container finished" podID="0c416919-9bad-4354-a369-433901bc43a7" containerID="810ffad5d6c2434b9e4134f42afba10aad6e31ea63433bb0959247643680d170" exitCode=0 Nov 02 22:53:29.354586 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:29.354448 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qgns6" event={"ID":"0c416919-9bad-4354-a369-433901bc43a7","Type":"ContainerDied","Data":"810ffad5d6c2434b9e4134f42afba10aad6e31ea63433bb0959247643680d170"} Nov 02 22:53:29.359375 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:29.359339 5727 generic.go:356] "Generic (PLEG): container finished" podID="495baab4-1e67-4202-8cb1-ab002c5dd19a" containerID="549b25ed837b6e98a4617d61709610871bd828264ee156b361d32fb84ad41ccc" exitCode=0 Nov 02 22:53:29.359462 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:29.359416 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7ngb8" event={"ID":"495baab4-1e67-4202-8cb1-ab002c5dd19a","Type":"ContainerDied","Data":"549b25ed837b6e98a4617d61709610871bd828264ee156b361d32fb84ad41ccc"} Nov 02 22:53:34.063783 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:34.063745 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 22:53:34.099081 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:34.098959 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 22:53:45.553226 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:45.553188 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c682r" event={"ID":"0ec8f93f-5c8f-4855-a174-96cad54fda07","Type":"ContainerStarted","Data":"bd3a8e73ab29a10b1ab0c8c968036eee4b2a0ca84ef1e101c988925e7a54c8cd"} Nov 02 22:53:46.577040 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:46.576382 5727 generic.go:356] "Generic (PLEG): container finished" podID="1389d802-5e2f-42df-b4b7-e10fccf8ceb7" containerID="49373c8ea888f819f28de4020f412bc3d514f25662f4cfbc7eeadfb842ed9c39" exitCode=0 Nov 02 22:53:46.577040 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:46.576520 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72f2rr" event={"ID":"1389d802-5e2f-42df-b4b7-e10fccf8ceb7","Type":"ContainerDied","Data":"49373c8ea888f819f28de4020f412bc3d514f25662f4cfbc7eeadfb842ed9c39"} Nov 02 22:53:46.581760 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:46.581462 5727 generic.go:356] "Generic (PLEG): container finished" podID="0ec8f93f-5c8f-4855-a174-96cad54fda07" containerID="bd3a8e73ab29a10b1ab0c8c968036eee4b2a0ca84ef1e101c988925e7a54c8cd" exitCode=0 Nov 02 22:53:46.581760 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:46.581542 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c682r" event={"ID":"0ec8f93f-5c8f-4855-a174-96cad54fda07","Type":"ContainerDied","Data":"bd3a8e73ab29a10b1ab0c8c968036eee4b2a0ca84ef1e101c988925e7a54c8cd"} Nov 02 22:53:46.586745 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:46.586168 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bf4w9" event={"ID":"56404571-73cf-4705-89d1-3450f088978f","Type":"ContainerStarted","Data":"e245f0954b57725d09b7ce3dd90f6d1ebd99b28908c65228b06d817baac6a8eb"} Nov 02 22:53:46.699457 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:46.698394 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c682r" podStartSLOduration=3.529353525 podStartE2EDuration="44.698375521s" totalImagesPullingTime="41.169021996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:53:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:45.673437227 +0000 UTC m=+311.911934925" watchObservedRunningTime="2026-11-02 22:53:46.698375521 +0000 UTC m=+312.936873184" Nov 02 22:53:46.935636 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:46.935569 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bf4w9" podStartSLOduration=2.834952591 podStartE2EDuration="42.935548577s" totalImagesPullingTime="40.100595986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:53:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:46.934374116 +0000 UTC m=+313.172871825" watchObservedRunningTime="2026-11-02 22:53:46.935548577 +0000 UTC m=+313.174046239" Nov 02 22:53:47.628856 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:47.627641 5727 generic.go:356] "Generic (PLEG): container finished" podID="56404571-73cf-4705-89d1-3450f088978f" containerID="e245f0954b57725d09b7ce3dd90f6d1ebd99b28908c65228b06d817baac6a8eb" exitCode=0 Nov 02 22:53:47.628856 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:47.628494 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bf4w9" event={"ID":"56404571-73cf-4705-89d1-3450f088978f","Type":"ContainerDied","Data":"e245f0954b57725d09b7ce3dd90f6d1ebd99b28908c65228b06d817baac6a8eb"} Nov 02 22:53:47.849220 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:47.849193 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72f2rr" Nov 02 22:53:47.907379 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:47.907307 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c682r" Nov 02 22:53:48.018156 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.017159 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ec8f93f-5c8f-4855-a174-96cad54fda07-kube-api-access-9mp9t\" (UniqueName: \"kubernetes.io/projected/0ec8f93f-5c8f-4855-a174-96cad54fda07-kube-api-access-9mp9t\") pod \"0ec8f93f-5c8f-4855-a174-96cad54fda07\" (UID: \"0ec8f93f-5c8f-4855-a174-96cad54fda07\") " Nov 02 22:53:48.018156 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.017260 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec8f93f-5c8f-4855-a174-96cad54fda07-util\" (UniqueName: \"kubernetes.io/empty-dir/0ec8f93f-5c8f-4855-a174-96cad54fda07-util\") pod \"0ec8f93f-5c8f-4855-a174-96cad54fda07\" (UID: \"0ec8f93f-5c8f-4855-a174-96cad54fda07\") " Nov 02 22:53:48.018156 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.017296 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1389d802-5e2f-42df-b4b7-e10fccf8ceb7-kube-api-access-b6wrp\" (UniqueName: \"kubernetes.io/projected/1389d802-5e2f-42df-b4b7-e10fccf8ceb7-kube-api-access-b6wrp\") pod \"1389d802-5e2f-42df-b4b7-e10fccf8ceb7\" (UID: \"1389d802-5e2f-42df-b4b7-e10fccf8ceb7\") " Nov 02 22:53:48.018156 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.017327 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1389d802-5e2f-42df-b4b7-e10fccf8ceb7-util\" (UniqueName: \"kubernetes.io/empty-dir/1389d802-5e2f-42df-b4b7-e10fccf8ceb7-util\") pod \"1389d802-5e2f-42df-b4b7-e10fccf8ceb7\" (UID: \"1389d802-5e2f-42df-b4b7-e10fccf8ceb7\") " Nov 02 22:53:48.018156 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.017427 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec8f93f-5c8f-4855-a174-96cad54fda07-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ec8f93f-5c8f-4855-a174-96cad54fda07-bundle\") pod \"0ec8f93f-5c8f-4855-a174-96cad54fda07\" (UID: \"0ec8f93f-5c8f-4855-a174-96cad54fda07\") " Nov 02 22:53:48.018156 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.017479 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1389d802-5e2f-42df-b4b7-e10fccf8ceb7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1389d802-5e2f-42df-b4b7-e10fccf8ceb7-bundle\") pod \"1389d802-5e2f-42df-b4b7-e10fccf8ceb7\" (UID: \"1389d802-5e2f-42df-b4b7-e10fccf8ceb7\") " Nov 02 22:53:48.020935 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.020622 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec8f93f-5c8f-4855-a174-96cad54fda07-bundle" pod "0ec8f93f-5c8f-4855-a174-96cad54fda07" (UID: "0ec8f93f-5c8f-4855-a174-96cad54fda07"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:48.022363 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.022316 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1389d802-5e2f-42df-b4b7-e10fccf8ceb7-bundle" pod "1389d802-5e2f-42df-b4b7-e10fccf8ceb7" (UID: "1389d802-5e2f-42df-b4b7-e10fccf8ceb7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:48.026411 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.026384 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec8f93f-5c8f-4855-a174-96cad54fda07-kube-api-access-9mp9t" pod "0ec8f93f-5c8f-4855-a174-96cad54fda07" (UID: "0ec8f93f-5c8f-4855-a174-96cad54fda07"). InnerVolumeSpecName "kube-api-access-9mp9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:53:48.029989 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.029967 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec8f93f-5c8f-4855-a174-96cad54fda07-util" pod "0ec8f93f-5c8f-4855-a174-96cad54fda07" (UID: "0ec8f93f-5c8f-4855-a174-96cad54fda07"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:48.030250 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.030224 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1389d802-5e2f-42df-b4b7-e10fccf8ceb7-util" pod "1389d802-5e2f-42df-b4b7-e10fccf8ceb7" (UID: "1389d802-5e2f-42df-b4b7-e10fccf8ceb7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:48.033338 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.033317 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1389d802-5e2f-42df-b4b7-e10fccf8ceb7-kube-api-access-b6wrp" pod "1389d802-5e2f-42df-b4b7-e10fccf8ceb7" (UID: "1389d802-5e2f-42df-b4b7-e10fccf8ceb7"). InnerVolumeSpecName "kube-api-access-b6wrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:53:48.118491 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.118449 5727 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ec8f93f-5c8f-4855-a174-96cad54fda07-bundle\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:48.118491 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.118480 5727 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1389d802-5e2f-42df-b4b7-e10fccf8ceb7-bundle\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:48.118674 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.118496 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mp9t\" (UniqueName: \"kubernetes.io/projected/0ec8f93f-5c8f-4855-a174-96cad54fda07-kube-api-access-9mp9t\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:48.118674 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.118510 5727 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ec8f93f-5c8f-4855-a174-96cad54fda07-util\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:48.118674 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.118524 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6wrp\" (UniqueName: \"kubernetes.io/projected/1389d802-5e2f-42df-b4b7-e10fccf8ceb7-kube-api-access-b6wrp\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:48.118674 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.118536 5727 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1389d802-5e2f-42df-b4b7-e10fccf8ceb7-util\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:48.639158 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.637406 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72f2rr" Nov 02 22:53:48.639643 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.639257 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72f2rr" event={"ID":"1389d802-5e2f-42df-b4b7-e10fccf8ceb7","Type":"ContainerDied","Data":"7df35813b05f1af874a6c6bb74dcfef70f0e8a89596fd01cf0d953f27b828459"} Nov 02 22:53:48.639643 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.639323 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7df35813b05f1af874a6c6bb74dcfef70f0e8a89596fd01cf0d953f27b828459" Nov 02 22:53:48.647692 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.647264 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv9n54" event={"ID":"6ba77a61-d033-4b7f-accf-2d842e49d934","Type":"ContainerStarted","Data":"c73e6220aec34166f16e2667ea6c381213a67ab0ede77dc813e3500e722f7776"} Nov 02 22:53:48.652493 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.652470 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c682r" Nov 02 22:53:48.653233 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.653201 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c682r" event={"ID":"0ec8f93f-5c8f-4855-a174-96cad54fda07","Type":"ContainerDied","Data":"42a29bca54932c3a793b99e07a299054e76335733b60da596c58d5cd9043ad26"} Nov 02 22:53:48.653354 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.653262 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42a29bca54932c3a793b99e07a299054e76335733b60da596c58d5cd9043ad26" Nov 02 22:53:48.773315 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.773248 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv9n54" podStartSLOduration=2.646218198 podStartE2EDuration="50.773229029s" totalImagesPullingTime="48.127010831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:52:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:48.770901644 +0000 UTC m=+315.009399308" watchObservedRunningTime="2026-11-02 22:53:48.773229029 +0000 UTC m=+315.011726692" Nov 02 22:53:48.884598 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.884577 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bf4w9" Nov 02 22:53:48.932730 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.932703 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56404571-73cf-4705-89d1-3450f088978f-util\" (UniqueName: \"kubernetes.io/empty-dir/56404571-73cf-4705-89d1-3450f088978f-util\") pod \"56404571-73cf-4705-89d1-3450f088978f\" (UID: \"56404571-73cf-4705-89d1-3450f088978f\") " Nov 02 22:53:48.932807 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.932759 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56404571-73cf-4705-89d1-3450f088978f-kube-api-access-7ts9w\" (UniqueName: \"kubernetes.io/projected/56404571-73cf-4705-89d1-3450f088978f-kube-api-access-7ts9w\") pod \"56404571-73cf-4705-89d1-3450f088978f\" (UID: \"56404571-73cf-4705-89d1-3450f088978f\") " Nov 02 22:53:48.932807 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.932794 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56404571-73cf-4705-89d1-3450f088978f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/56404571-73cf-4705-89d1-3450f088978f-bundle\") pod \"56404571-73cf-4705-89d1-3450f088978f\" (UID: \"56404571-73cf-4705-89d1-3450f088978f\") " Nov 02 22:53:48.935356 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.935322 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56404571-73cf-4705-89d1-3450f088978f-bundle" pod "56404571-73cf-4705-89d1-3450f088978f" (UID: "56404571-73cf-4705-89d1-3450f088978f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:48.940852 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.940809 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56404571-73cf-4705-89d1-3450f088978f-kube-api-access-7ts9w" pod "56404571-73cf-4705-89d1-3450f088978f" (UID: "56404571-73cf-4705-89d1-3450f088978f"). InnerVolumeSpecName "kube-api-access-7ts9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:53:48.941641 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:48.941613 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56404571-73cf-4705-89d1-3450f088978f-util" pod "56404571-73cf-4705-89d1-3450f088978f" (UID: "56404571-73cf-4705-89d1-3450f088978f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:49.033809 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:49.033579 5727 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56404571-73cf-4705-89d1-3450f088978f-util\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:49.033809 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:49.033610 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ts9w\" (UniqueName: \"kubernetes.io/projected/56404571-73cf-4705-89d1-3450f088978f-kube-api-access-7ts9w\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:49.033809 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:49.033630 5727 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56404571-73cf-4705-89d1-3450f088978f-bundle\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:49.661592 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:49.661512 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvtmn" event={"ID":"8559b43f-b9d0-4fed-90a0-c01922032b21","Type":"ContainerStarted","Data":"917a5d3b449ee955550beba2d093f0890d2aaa4379bcda2594f47d10d8d9b7f3"} Nov 02 22:53:49.664133 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:49.664102 5727 generic.go:356] "Generic (PLEG): container finished" podID="6ba77a61-d033-4b7f-accf-2d842e49d934" containerID="c73e6220aec34166f16e2667ea6c381213a67ab0ede77dc813e3500e722f7776" exitCode=0 Nov 02 22:53:49.664260 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:49.664165 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv9n54" event={"ID":"6ba77a61-d033-4b7f-accf-2d842e49d934","Type":"ContainerDied","Data":"c73e6220aec34166f16e2667ea6c381213a67ab0ede77dc813e3500e722f7776"} Nov 02 22:53:49.666157 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:49.666117 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 02 22:53:49.666863 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:49.666840 5727 generic.go:356] "Generic (PLEG): container finished" podID="d9ba2c71-9d0f-41f8-8f4c-b3b510183df3" containerID="3ec6a646320642b717b93ac15beebf1460d152e3eeaf2112b0a2183320d97712" exitCode=1 Nov 02 22:53:49.666991 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:49.666939 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn" event={"ID":"d9ba2c71-9d0f-41f8-8f4c-b3b510183df3","Type":"ContainerDied","Data":"3ec6a646320642b717b93ac15beebf1460d152e3eeaf2112b0a2183320d97712"} Nov 02 22:53:49.669688 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:49.669668 5727 generic.go:356] "Generic (PLEG): container finished" podID="9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c" containerID="595c61fdf2950150aee4a7191de79976599331c06b56c619270eb9fb87ee0d75" exitCode=0 Nov 02 22:53:49.669810 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:49.669762 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67265n" event={"ID":"9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c","Type":"ContainerDied","Data":"595c61fdf2950150aee4a7191de79976599331c06b56c619270eb9fb87ee0d75"} Nov 02 22:53:49.672354 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:49.672329 5727 generic.go:356] "Generic (PLEG): container finished" podID="26f14f8d-74a6-4c09-865b-b10bd97c4fb2" containerID="56682fe327369a64bc16da1634990dbb5fbf6f32e1341db63dfc551ff06d59f9" exitCode=0 Nov 02 22:53:49.672446 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:49.672404 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxhx76" event={"ID":"26f14f8d-74a6-4c09-865b-b10bd97c4fb2","Type":"ContainerDied","Data":"56682fe327369a64bc16da1634990dbb5fbf6f32e1341db63dfc551ff06d59f9"} Nov 02 22:53:49.674645 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:49.674616 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bf4w9" event={"ID":"56404571-73cf-4705-89d1-3450f088978f","Type":"ContainerDied","Data":"8bb45241d25967fb8b4d3b51b273de17da398f4a9cacc3479fb708680f633891"} Nov 02 22:53:49.674741 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:49.674672 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bb45241d25967fb8b4d3b51b273de17da398f4a9cacc3479fb708680f633891" Nov 02 22:53:49.674741 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:49.674628 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bf4w9" Nov 02 22:53:49.676787 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:49.676764 5727 generic.go:356] "Generic (PLEG): container finished" podID="83f5166e-d9cc-45b1-9c5c-9189b8de96df" containerID="5be04586460c3a101d46f0eb460ce7db8bd4f5459713a5456d4ce48867161bc9" exitCode=0 Nov 02 22:53:49.676865 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:49.676798 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5nvcz" event={"ID":"83f5166e-d9cc-45b1-9c5c-9189b8de96df","Type":"ContainerDied","Data":"5be04586460c3a101d46f0eb460ce7db8bd4f5459713a5456d4ce48867161bc9"} Nov 02 22:53:49.913990 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:49.913886 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvtmn" podStartSLOduration=3.043546361 podStartE2EDuration="51.913871302s" totalImagesPullingTime="48.870324941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:52:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:49.791430306 +0000 UTC m=+316.029928030" watchObservedRunningTime="2026-11-02 22:53:49.913871302 +0000 UTC m=+316.152369010" Nov 02 22:53:50.684366 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.684340 5727 generic.go:356] "Generic (PLEG): container finished" podID="8559b43f-b9d0-4fed-90a0-c01922032b21" containerID="917a5d3b449ee955550beba2d093f0890d2aaa4379bcda2594f47d10d8d9b7f3" exitCode=0 Nov 02 22:53:50.684720 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.684420 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvtmn" event={"ID":"8559b43f-b9d0-4fed-90a0-c01922032b21","Type":"ContainerDied","Data":"917a5d3b449ee955550beba2d093f0890d2aaa4379bcda2594f47d10d8d9b7f3"} Nov 02 22:53:50.686572 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.686554 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7ngb8" event={"ID":"495baab4-1e67-4202-8cb1-ab002c5dd19a","Type":"ContainerStarted","Data":"d28bbe4ab4584348ab937322b4e07f4a31014350fc2cc2ef90aa2b1c75d412ea"} Nov 02 22:53:50.688540 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.688520 5727 generic.go:356] "Generic (PLEG): container finished" podID="0c416919-9bad-4354-a369-433901bc43a7" containerID="76689e3816f795ca533c5b6ff46cc14a4ec9ac7fc2b444a1fb565ee54e6f7bde" exitCode=0 Nov 02 22:53:50.688651 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.688601 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qgns6" event={"ID":"0c416919-9bad-4354-a369-433901bc43a7","Type":"ContainerDied","Data":"76689e3816f795ca533c5b6ff46cc14a4ec9ac7fc2b444a1fb565ee54e6f7bde"} Nov 02 22:53:50.853523 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.853496 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxhx76" Nov 02 22:53:50.890852 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.890827 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 02 22:53:50.893655 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.893634 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn" Nov 02 22:53:50.896455 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.896433 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67265n" Nov 02 22:53:50.924703 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.924685 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5nvcz" Nov 02 22:53:50.931269 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.931249 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv9n54" Nov 02 22:53:50.952599 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.952578 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c-bundle\") pod \"9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c\" (UID: \"9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c\") " Nov 02 22:53:50.952713 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.952619 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba77a61-d033-4b7f-accf-2d842e49d934-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ba77a61-d033-4b7f-accf-2d842e49d934-bundle\") pod \"6ba77a61-d033-4b7f-accf-2d842e49d934\" (UID: \"6ba77a61-d033-4b7f-accf-2d842e49d934\") " Nov 02 22:53:50.952713 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.952642 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83f5166e-d9cc-45b1-9c5c-9189b8de96df-kube-api-access-fcwff\" (UniqueName: \"kubernetes.io/projected/83f5166e-d9cc-45b1-9c5c-9189b8de96df-kube-api-access-fcwff\") pod \"83f5166e-d9cc-45b1-9c5c-9189b8de96df\" (UID: \"83f5166e-d9cc-45b1-9c5c-9189b8de96df\") " Nov 02 22:53:50.952713 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.952676 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c-util\" (UniqueName: \"kubernetes.io/empty-dir/9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c-util\") pod \"9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c\" (UID: \"9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c\") " Nov 02 22:53:50.952713 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.952694 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ba77a61-d033-4b7f-accf-2d842e49d934-kube-api-access-7q2l7\" (UniqueName: \"kubernetes.io/projected/6ba77a61-d033-4b7f-accf-2d842e49d934-kube-api-access-7q2l7\") pod \"6ba77a61-d033-4b7f-accf-2d842e49d934\" (UID: \"6ba77a61-d033-4b7f-accf-2d842e49d934\") " Nov 02 22:53:50.952713 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.952709 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba77a61-d033-4b7f-accf-2d842e49d934-util\" (UniqueName: \"kubernetes.io/empty-dir/6ba77a61-d033-4b7f-accf-2d842e49d934-util\") pod \"6ba77a61-d033-4b7f-accf-2d842e49d934\" (UID: \"6ba77a61-d033-4b7f-accf-2d842e49d934\") " Nov 02 22:53:50.952964 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.952754 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ba2c71-9d0f-41f8-8f4c-b3b510183df3-util\" (UniqueName: \"kubernetes.io/empty-dir/d9ba2c71-9d0f-41f8-8f4c-b3b510183df3-util\") pod \"d9ba2c71-9d0f-41f8-8f4c-b3b510183df3\" (UID: \"d9ba2c71-9d0f-41f8-8f4c-b3b510183df3\") " Nov 02 22:53:50.952964 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.952781 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83f5166e-d9cc-45b1-9c5c-9189b8de96df-util\" (UniqueName: \"kubernetes.io/empty-dir/83f5166e-d9cc-45b1-9c5c-9189b8de96df-util\") pod \"83f5166e-d9cc-45b1-9c5c-9189b8de96df\" (UID: \"83f5166e-d9cc-45b1-9c5c-9189b8de96df\") " Nov 02 22:53:50.952964 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.952807 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f14f8d-74a6-4c09-865b-b10bd97c4fb2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/26f14f8d-74a6-4c09-865b-b10bd97c4fb2-bundle\") pod \"26f14f8d-74a6-4c09-865b-b10bd97c4fb2\" (UID: \"26f14f8d-74a6-4c09-865b-b10bd97c4fb2\") " Nov 02 22:53:50.952964 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.952836 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f14f8d-74a6-4c09-865b-b10bd97c4fb2-util\" (UniqueName: \"kubernetes.io/empty-dir/26f14f8d-74a6-4c09-865b-b10bd97c4fb2-util\") pod \"26f14f8d-74a6-4c09-865b-b10bd97c4fb2\" (UID: \"26f14f8d-74a6-4c09-865b-b10bd97c4fb2\") " Nov 02 22:53:50.952964 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.952890 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c-kube-api-access-f5xk7\" (UniqueName: \"kubernetes.io/projected/9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c-kube-api-access-f5xk7\") pod \"9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c\" (UID: \"9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c\") " Nov 02 22:53:50.952964 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.952930 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83f5166e-d9cc-45b1-9c5c-9189b8de96df-bundle\" (UniqueName: \"kubernetes.io/empty-dir/83f5166e-d9cc-45b1-9c5c-9189b8de96df-bundle\") pod \"83f5166e-d9cc-45b1-9c5c-9189b8de96df\" (UID: \"83f5166e-d9cc-45b1-9c5c-9189b8de96df\") " Nov 02 22:53:50.953301 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.952965 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9ba2c71-9d0f-41f8-8f4c-b3b510183df3-kube-api-access-7n962\" (UniqueName: \"kubernetes.io/projected/d9ba2c71-9d0f-41f8-8f4c-b3b510183df3-kube-api-access-7n962\") pod \"d9ba2c71-9d0f-41f8-8f4c-b3b510183df3\" (UID: \"d9ba2c71-9d0f-41f8-8f4c-b3b510183df3\") " Nov 02 22:53:50.953301 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.952995 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f14f8d-74a6-4c09-865b-b10bd97c4fb2-kube-api-access-v62jh\" (UniqueName: \"kubernetes.io/projected/26f14f8d-74a6-4c09-865b-b10bd97c4fb2-kube-api-access-v62jh\") pod \"26f14f8d-74a6-4c09-865b-b10bd97c4fb2\" (UID: \"26f14f8d-74a6-4c09-865b-b10bd97c4fb2\") " Nov 02 22:53:50.953301 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.953034 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ba2c71-9d0f-41f8-8f4c-b3b510183df3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9ba2c71-9d0f-41f8-8f4c-b3b510183df3-bundle\") pod \"d9ba2c71-9d0f-41f8-8f4c-b3b510183df3\" (UID: \"d9ba2c71-9d0f-41f8-8f4c-b3b510183df3\") " Nov 02 22:53:50.953301 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.953193 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c-bundle" pod "9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c" (UID: "9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:50.953301 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.953247 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba77a61-d033-4b7f-accf-2d842e49d934-bundle" pod "6ba77a61-d033-4b7f-accf-2d842e49d934" (UID: "6ba77a61-d033-4b7f-accf-2d842e49d934"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:50.953544 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.953312 5727 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c-bundle\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:50.953544 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.953331 5727 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ba77a61-d033-4b7f-accf-2d842e49d934-bundle\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:50.954146 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.954099 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f14f8d-74a6-4c09-865b-b10bd97c4fb2-bundle" pod "26f14f8d-74a6-4c09-865b-b10bd97c4fb2" (UID: "26f14f8d-74a6-4c09-865b-b10bd97c4fb2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:50.954431 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.954406 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83f5166e-d9cc-45b1-9c5c-9189b8de96df-bundle" pod "83f5166e-d9cc-45b1-9c5c-9189b8de96df" (UID: "83f5166e-d9cc-45b1-9c5c-9189b8de96df"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:50.955667 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.955638 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba77a61-d033-4b7f-accf-2d842e49d934-kube-api-access-7q2l7" pod "6ba77a61-d033-4b7f-accf-2d842e49d934" (UID: "6ba77a61-d033-4b7f-accf-2d842e49d934"). InnerVolumeSpecName "kube-api-access-7q2l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:53:50.955954 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.955931 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c-kube-api-access-f5xk7" pod "9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c" (UID: "9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c"). InnerVolumeSpecName "kube-api-access-f5xk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:53:50.956045 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.955974 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83f5166e-d9cc-45b1-9c5c-9189b8de96df-kube-api-access-fcwff" pod "83f5166e-d9cc-45b1-9c5c-9189b8de96df" (UID: "83f5166e-d9cc-45b1-9c5c-9189b8de96df"). InnerVolumeSpecName "kube-api-access-fcwff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:53:50.956638 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.956600 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ba2c71-9d0f-41f8-8f4c-b3b510183df3-bundle" pod "d9ba2c71-9d0f-41f8-8f4c-b3b510183df3" (UID: "d9ba2c71-9d0f-41f8-8f4c-b3b510183df3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:50.956886 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.956852 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ba2c71-9d0f-41f8-8f4c-b3b510183df3-kube-api-access-7n962" pod "d9ba2c71-9d0f-41f8-8f4c-b3b510183df3" (UID: "d9ba2c71-9d0f-41f8-8f4c-b3b510183df3"). InnerVolumeSpecName "kube-api-access-7n962". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:53:50.957869 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.957849 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f14f8d-74a6-4c09-865b-b10bd97c4fb2-kube-api-access-v62jh" pod "26f14f8d-74a6-4c09-865b-b10bd97c4fb2" (UID: "26f14f8d-74a6-4c09-865b-b10bd97c4fb2"). InnerVolumeSpecName "kube-api-access-v62jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:53:50.963136 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.963101 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c-util" pod "9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c" (UID: "9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:50.963228 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.963202 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ba2c71-9d0f-41f8-8f4c-b3b510183df3-util" pod "d9ba2c71-9d0f-41f8-8f4c-b3b510183df3" (UID: "d9ba2c71-9d0f-41f8-8f4c-b3b510183df3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:50.964291 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.964274 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83f5166e-d9cc-45b1-9c5c-9189b8de96df-util" pod "83f5166e-d9cc-45b1-9c5c-9189b8de96df" (UID: "83f5166e-d9cc-45b1-9c5c-9189b8de96df"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:50.965010 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.964981 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba77a61-d033-4b7f-accf-2d842e49d934-util" pod "6ba77a61-d033-4b7f-accf-2d842e49d934" (UID: "6ba77a61-d033-4b7f-accf-2d842e49d934"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:50.965786 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:50.965767 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f14f8d-74a6-4c09-865b-b10bd97c4fb2-util" pod "26f14f8d-74a6-4c09-865b-b10bd97c4fb2" (UID: "26f14f8d-74a6-4c09-865b-b10bd97c4fb2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:51.048988 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.048943 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7ngb8" podStartSLOduration=4.769940601 podStartE2EDuration="54.048929229s" totalImagesPullingTime="49.278988628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:52:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:50.929416266 +0000 UTC m=+317.167913939" watchObservedRunningTime="2026-11-02 22:53:51.048929229 +0000 UTC m=+317.287426941" Nov 02 22:53:51.054175 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.054154 5727 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9ba2c71-9d0f-41f8-8f4c-b3b510183df3-util\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:51.054175 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.054174 5727 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83f5166e-d9cc-45b1-9c5c-9189b8de96df-util\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:51.054300 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.054183 5727 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26f14f8d-74a6-4c09-865b-b10bd97c4fb2-bundle\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:51.054300 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.054192 5727 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26f14f8d-74a6-4c09-865b-b10bd97c4fb2-util\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:51.054300 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.054201 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5xk7\" (UniqueName: \"kubernetes.io/projected/9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c-kube-api-access-f5xk7\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:51.054300 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.054210 5727 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83f5166e-d9cc-45b1-9c5c-9189b8de96df-bundle\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:51.054300 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.054219 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n962\" (UniqueName: \"kubernetes.io/projected/d9ba2c71-9d0f-41f8-8f4c-b3b510183df3-kube-api-access-7n962\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:51.054300 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.054227 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v62jh\" (UniqueName: \"kubernetes.io/projected/26f14f8d-74a6-4c09-865b-b10bd97c4fb2-kube-api-access-v62jh\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:51.054300 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.054235 5727 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9ba2c71-9d0f-41f8-8f4c-b3b510183df3-bundle\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:51.054300 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.054244 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcwff\" (UniqueName: \"kubernetes.io/projected/83f5166e-d9cc-45b1-9c5c-9189b8de96df-kube-api-access-fcwff\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:51.054300 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.054252 5727 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c-util\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:51.054300 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.054260 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q2l7\" (UniqueName: \"kubernetes.io/projected/6ba77a61-d033-4b7f-accf-2d842e49d934-kube-api-access-7q2l7\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:51.054300 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.054268 5727 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ba77a61-d033-4b7f-accf-2d842e49d934-util\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:51.695595 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.695562 5727 generic.go:356] "Generic (PLEG): container finished" podID="495baab4-1e67-4202-8cb1-ab002c5dd19a" containerID="d28bbe4ab4584348ab937322b4e07f4a31014350fc2cc2ef90aa2b1c75d412ea" exitCode=0 Nov 02 22:53:51.696013 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.695651 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7ngb8" event={"ID":"495baab4-1e67-4202-8cb1-ab002c5dd19a","Type":"ContainerDied","Data":"d28bbe4ab4584348ab937322b4e07f4a31014350fc2cc2ef90aa2b1c75d412ea"} Nov 02 22:53:51.697584 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.697562 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv9n54" Nov 02 22:53:51.697584 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.697577 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv9n54" event={"ID":"6ba77a61-d033-4b7f-accf-2d842e49d934","Type":"ContainerDied","Data":"d811225ea4455791d224a80f68ba12635696267c77c6dff0a0fd8d3ee86afddc"} Nov 02 22:53:51.697751 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.697611 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d811225ea4455791d224a80f68ba12635696267c77c6dff0a0fd8d3ee86afddc" Nov 02 22:53:51.699011 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.698990 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 02 22:53:51.699740 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.699718 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn" Nov 02 22:53:51.699830 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.699722 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn" event={"ID":"d9ba2c71-9d0f-41f8-8f4c-b3b510183df3","Type":"ContainerDied","Data":"dd7bca809dd188377c6baf4e004ed71c64f7181497ff2dcb567686e1755d2f9f"} Nov 02 22:53:51.699897 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.699842 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd7bca809dd188377c6baf4e004ed71c64f7181497ff2dcb567686e1755d2f9f" Nov 02 22:53:51.701783 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.701765 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67265n" Nov 02 22:53:51.701884 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.701768 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67265n" event={"ID":"9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c","Type":"ContainerDied","Data":"e2b32eccdbc2340d1439836282bdd96f753d956121733f3000043b7eaf53bef5"} Nov 02 22:53:51.701884 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.701873 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2b32eccdbc2340d1439836282bdd96f753d956121733f3000043b7eaf53bef5" Nov 02 22:53:51.703879 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.703864 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxhx76" Nov 02 22:53:51.703972 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.703929 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxhx76" event={"ID":"26f14f8d-74a6-4c09-865b-b10bd97c4fb2","Type":"ContainerDied","Data":"f62c89e158f132812efe1eb7279adf08e2fbe96bf40fad05214d7a9bc7b5293c"} Nov 02 22:53:51.704040 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.703981 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f62c89e158f132812efe1eb7279adf08e2fbe96bf40fad05214d7a9bc7b5293c" Nov 02 22:53:51.706206 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.706188 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5nvcz" Nov 02 22:53:51.707049 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.706323 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5nvcz" event={"ID":"83f5166e-d9cc-45b1-9c5c-9189b8de96df","Type":"ContainerDied","Data":"ff412771ba5ce1abcb4e0209a106d7a7e9d7d4d1816004ab177d0f34bb8d2263"} Nov 02 22:53:51.707049 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.706358 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff412771ba5ce1abcb4e0209a106d7a7e9d7d4d1816004ab177d0f34bb8d2263" Nov 02 22:53:51.707702 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707189 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-pd79m"] Nov 02 22:53:51.707702 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707641 5727 cpu_manager.go:414] "containerMap: removing container" podUID="0ec8f93f-5c8f-4855-a174-96cad54fda07" containerName="util" Nov 02 22:53:51.707702 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707657 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ec8f93f-5c8f-4855-a174-96cad54fda07" containerName="util" Nov 02 22:53:51.707702 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707672 5727 cpu_manager.go:414] "containerMap: removing container" podUID="0ec8f93f-5c8f-4855-a174-96cad54fda07" containerName="extract" Nov 02 22:53:51.707702 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707679 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ec8f93f-5c8f-4855-a174-96cad54fda07" containerName="extract" Nov 02 22:53:51.707702 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707688 5727 cpu_manager.go:414] "containerMap: removing container" podUID="83f5166e-d9cc-45b1-9c5c-9189b8de96df" containerName="extract" Nov 02 22:53:51.707702 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707696 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83f5166e-d9cc-45b1-9c5c-9189b8de96df" containerName="extract" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707712 5727 cpu_manager.go:414] "containerMap: removing container" podUID="26f14f8d-74a6-4c09-865b-b10bd97c4fb2" containerName="pull" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707719 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f14f8d-74a6-4c09-865b-b10bd97c4fb2" containerName="pull" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707731 5727 cpu_manager.go:414] "containerMap: removing container" podUID="83f5166e-d9cc-45b1-9c5c-9189b8de96df" containerName="util" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707738 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83f5166e-d9cc-45b1-9c5c-9189b8de96df" containerName="util" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707747 5727 cpu_manager.go:414] "containerMap: removing container" podUID="56404571-73cf-4705-89d1-3450f088978f" containerName="util" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707755 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56404571-73cf-4705-89d1-3450f088978f" containerName="util" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707768 5727 cpu_manager.go:414] "containerMap: removing container" podUID="d9ba2c71-9d0f-41f8-8f4c-b3b510183df3" containerName="pull" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707775 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9ba2c71-9d0f-41f8-8f4c-b3b510183df3" containerName="pull" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707789 5727 cpu_manager.go:414] "containerMap: removing container" podUID="9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c" containerName="extract" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707795 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c" containerName="extract" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707815 5727 cpu_manager.go:414] "containerMap: removing container" podUID="26f14f8d-74a6-4c09-865b-b10bd97c4fb2" containerName="util" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707822 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f14f8d-74a6-4c09-865b-b10bd97c4fb2" containerName="util" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707831 5727 cpu_manager.go:414] "containerMap: removing container" podUID="9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c" containerName="pull" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707837 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c" containerName="pull" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707847 5727 cpu_manager.go:414] "containerMap: removing container" podUID="83f5166e-d9cc-45b1-9c5c-9189b8de96df" containerName="pull" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707853 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83f5166e-d9cc-45b1-9c5c-9189b8de96df" containerName="pull" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707863 5727 cpu_manager.go:414] "containerMap: removing container" podUID="9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c" containerName="util" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707869 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c" containerName="util" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707879 5727 cpu_manager.go:414] "containerMap: removing container" podUID="56404571-73cf-4705-89d1-3450f088978f" containerName="extract" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707886 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56404571-73cf-4705-89d1-3450f088978f" containerName="extract" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707893 5727 cpu_manager.go:414] "containerMap: removing container" podUID="26f14f8d-74a6-4c09-865b-b10bd97c4fb2" containerName="extract" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707900 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f14f8d-74a6-4c09-865b-b10bd97c4fb2" containerName="extract" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707908 5727 cpu_manager.go:414] "containerMap: removing container" podUID="1389d802-5e2f-42df-b4b7-e10fccf8ceb7" containerName="pull" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707914 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1389d802-5e2f-42df-b4b7-e10fccf8ceb7" containerName="pull" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707922 5727 cpu_manager.go:414] "containerMap: removing container" podUID="56404571-73cf-4705-89d1-3450f088978f" containerName="pull" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707929 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56404571-73cf-4705-89d1-3450f088978f" containerName="pull" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707941 5727 cpu_manager.go:414] "containerMap: removing container" podUID="1389d802-5e2f-42df-b4b7-e10fccf8ceb7" containerName="util" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707947 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1389d802-5e2f-42df-b4b7-e10fccf8ceb7" containerName="util" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707957 5727 cpu_manager.go:414] "containerMap: removing container" podUID="d9ba2c71-9d0f-41f8-8f4c-b3b510183df3" containerName="util" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707965 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9ba2c71-9d0f-41f8-8f4c-b3b510183df3" containerName="util" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707973 5727 cpu_manager.go:414] "containerMap: removing container" podUID="6ba77a61-d033-4b7f-accf-2d842e49d934" containerName="pull" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707980 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ba77a61-d033-4b7f-accf-2d842e49d934" containerName="pull" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707989 5727 cpu_manager.go:414] "containerMap: removing container" podUID="6ba77a61-d033-4b7f-accf-2d842e49d934" containerName="extract" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.707998 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ba77a61-d033-4b7f-accf-2d842e49d934" containerName="extract" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.708016 5727 cpu_manager.go:414] "containerMap: removing container" podUID="0ec8f93f-5c8f-4855-a174-96cad54fda07" containerName="pull" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.708023 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ec8f93f-5c8f-4855-a174-96cad54fda07" containerName="pull" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.708037 5727 cpu_manager.go:414] "containerMap: removing container" podUID="1389d802-5e2f-42df-b4b7-e10fccf8ceb7" containerName="extract" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.708044 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1389d802-5e2f-42df-b4b7-e10fccf8ceb7" containerName="extract" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.708056 5727 cpu_manager.go:414] "containerMap: removing container" podUID="6ba77a61-d033-4b7f-accf-2d842e49d934" containerName="util" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.708062 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ba77a61-d033-4b7f-accf-2d842e49d934" containerName="util" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.708073 5727 cpu_manager.go:414] "containerMap: removing container" podUID="d9ba2c71-9d0f-41f8-8f4c-b3b510183df3" containerName="extract" Nov 02 22:53:51.708147 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.708080 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9ba2c71-9d0f-41f8-8f4c-b3b510183df3" containerName="extract" Nov 02 22:53:51.709797 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.708192 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ba77a61-d033-4b7f-accf-2d842e49d934" containerName="extract" Nov 02 22:53:51.709797 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.708206 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="1389d802-5e2f-42df-b4b7-e10fccf8ceb7" containerName="extract" Nov 02 22:53:51.709797 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.708215 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9ba2c71-9d0f-41f8-8f4c-b3b510183df3" containerName="extract" Nov 02 22:53:51.709797 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.708223 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ec8f93f-5c8f-4855-a174-96cad54fda07" containerName="extract" Nov 02 22:53:51.709797 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.708232 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="83f5166e-d9cc-45b1-9c5c-9189b8de96df" containerName="extract" Nov 02 22:53:51.709797 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.708244 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a110d03-4fe0-4d47-b9c0-fd1ca3ac180c" containerName="extract" Nov 02 22:53:51.709797 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.708255 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="56404571-73cf-4705-89d1-3450f088978f" containerName="extract" Nov 02 22:53:51.709797 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.708263 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f14f8d-74a6-4c09-865b-b10bd97c4fb2" containerName="extract" Nov 02 22:53:51.711635 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.711610 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-pd79m" Nov 02 22:53:51.759540 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.759491 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ab5f727-4be4-466f-8cbc-10874091208d-metrics-cert\") pod \"opendatahub-operator-controller-manager-79d467458d-pd79m\" (UID: \"2ab5f727-4be4-466f-8cbc-10874091208d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-pd79m" Nov 02 22:53:51.759683 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.759570 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2ab5f727-4be4-466f-8cbc-10874091208d-manifests\") pod \"opendatahub-operator-controller-manager-79d467458d-pd79m\" (UID: \"2ab5f727-4be4-466f-8cbc-10874091208d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-pd79m" Nov 02 22:53:51.759720 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.759678 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tct9b\" (UniqueName: \"kubernetes.io/projected/2ab5f727-4be4-466f-8cbc-10874091208d-kube-api-access-tct9b\") pod \"opendatahub-operator-controller-manager-79d467458d-pd79m\" (UID: \"2ab5f727-4be4-466f-8cbc-10874091208d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-pd79m" Nov 02 22:53:51.759720 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.759709 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ab5f727-4be4-466f-8cbc-10874091208d-webhook-cert\") pod \"opendatahub-operator-controller-manager-79d467458d-pd79m\" (UID: \"2ab5f727-4be4-466f-8cbc-10874091208d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-pd79m" Nov 02 22:53:51.759801 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.759783 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2ab5f727-4be4-466f-8cbc-10874091208d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79d467458d-pd79m\" (UID: \"2ab5f727-4be4-466f-8cbc-10874091208d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-pd79m" Nov 02 22:53:51.759848 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.759808 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ab5f727-4be4-466f-8cbc-10874091208d-tmp\") pod \"opendatahub-operator-controller-manager-79d467458d-pd79m\" (UID: \"2ab5f727-4be4-466f-8cbc-10874091208d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-pd79m" Nov 02 22:53:51.774792 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.774764 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 22:53:51.774910 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.774823 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 22:53:51.774910 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.774865 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 22:53:51.774996 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.774952 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 22:53:51.775064 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.775047 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-c5ts9\"" Nov 02 22:53:51.775103 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.775090 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 22:53:51.860215 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.860192 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ab5f727-4be4-466f-8cbc-10874091208d-tmp\") pod \"opendatahub-operator-controller-manager-79d467458d-pd79m\" (UID: \"2ab5f727-4be4-466f-8cbc-10874091208d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-pd79m" Nov 02 22:53:51.860326 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.860264 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ab5f727-4be4-466f-8cbc-10874091208d-metrics-cert\") pod \"opendatahub-operator-controller-manager-79d467458d-pd79m\" (UID: \"2ab5f727-4be4-466f-8cbc-10874091208d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-pd79m" Nov 02 22:53:51.860326 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.860288 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2ab5f727-4be4-466f-8cbc-10874091208d-manifests\") pod \"opendatahub-operator-controller-manager-79d467458d-pd79m\" (UID: \"2ab5f727-4be4-466f-8cbc-10874091208d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-pd79m" Nov 02 22:53:51.860458 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.860348 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tct9b\" (UniqueName: \"kubernetes.io/projected/2ab5f727-4be4-466f-8cbc-10874091208d-kube-api-access-tct9b\") pod \"opendatahub-operator-controller-manager-79d467458d-pd79m\" (UID: \"2ab5f727-4be4-466f-8cbc-10874091208d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-pd79m" Nov 02 22:53:51.860458 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.860380 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ab5f727-4be4-466f-8cbc-10874091208d-webhook-cert\") pod \"opendatahub-operator-controller-manager-79d467458d-pd79m\" (UID: \"2ab5f727-4be4-466f-8cbc-10874091208d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-pd79m" Nov 02 22:53:51.860458 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.860424 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2ab5f727-4be4-466f-8cbc-10874091208d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79d467458d-pd79m\" (UID: \"2ab5f727-4be4-466f-8cbc-10874091208d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-pd79m" Nov 02 22:53:51.860731 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.860705 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2ab5f727-4be4-466f-8cbc-10874091208d-manifests\") pod \"opendatahub-operator-controller-manager-79d467458d-pd79m\" (UID: \"2ab5f727-4be4-466f-8cbc-10874091208d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-pd79m" Nov 02 22:53:51.862530 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.862505 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qgns6" Nov 02 22:53:51.862846 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.862819 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ab5f727-4be4-466f-8cbc-10874091208d-tmp\") pod \"opendatahub-operator-controller-manager-79d467458d-pd79m\" (UID: \"2ab5f727-4be4-466f-8cbc-10874091208d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-pd79m" Nov 02 22:53:51.862957 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.862844 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2ab5f727-4be4-466f-8cbc-10874091208d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79d467458d-pd79m\" (UID: \"2ab5f727-4be4-466f-8cbc-10874091208d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-pd79m" Nov 02 22:53:51.863068 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.863051 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ab5f727-4be4-466f-8cbc-10874091208d-metrics-cert\") pod \"opendatahub-operator-controller-manager-79d467458d-pd79m\" (UID: \"2ab5f727-4be4-466f-8cbc-10874091208d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-pd79m" Nov 02 22:53:51.863608 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.863588 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ab5f727-4be4-466f-8cbc-10874091208d-webhook-cert\") pod \"opendatahub-operator-controller-manager-79d467458d-pd79m\" (UID: \"2ab5f727-4be4-466f-8cbc-10874091208d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-pd79m" Nov 02 22:53:51.882741 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.878459 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvtmn" Nov 02 22:53:51.887325 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.887306 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-pd79m"] Nov 02 22:53:51.920602 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.920575 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tct9b\" (UniqueName: \"kubernetes.io/projected/2ab5f727-4be4-466f-8cbc-10874091208d-kube-api-access-tct9b\") pod \"opendatahub-operator-controller-manager-79d467458d-pd79m\" (UID: \"2ab5f727-4be4-466f-8cbc-10874091208d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-pd79m" Nov 02 22:53:51.961224 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.961176 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8559b43f-b9d0-4fed-90a0-c01922032b21-kube-api-access-bj58t\" (UniqueName: \"kubernetes.io/projected/8559b43f-b9d0-4fed-90a0-c01922032b21-kube-api-access-bj58t\") pod \"8559b43f-b9d0-4fed-90a0-c01922032b21\" (UID: \"8559b43f-b9d0-4fed-90a0-c01922032b21\") " Nov 02 22:53:51.961348 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.961232 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c416919-9bad-4354-a369-433901bc43a7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c416919-9bad-4354-a369-433901bc43a7-bundle\") pod \"0c416919-9bad-4354-a369-433901bc43a7\" (UID: \"0c416919-9bad-4354-a369-433901bc43a7\") " Nov 02 22:53:51.961348 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.961249 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c416919-9bad-4354-a369-433901bc43a7-util\" (UniqueName: \"kubernetes.io/empty-dir/0c416919-9bad-4354-a369-433901bc43a7-util\") pod \"0c416919-9bad-4354-a369-433901bc43a7\" (UID: \"0c416919-9bad-4354-a369-433901bc43a7\") " Nov 02 22:53:51.961472 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.961370 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c416919-9bad-4354-a369-433901bc43a7-kube-api-access-lq64r\" (UniqueName: \"kubernetes.io/projected/0c416919-9bad-4354-a369-433901bc43a7-kube-api-access-lq64r\") pod \"0c416919-9bad-4354-a369-433901bc43a7\" (UID: \"0c416919-9bad-4354-a369-433901bc43a7\") " Nov 02 22:53:51.961531 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.961472 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8559b43f-b9d0-4fed-90a0-c01922032b21-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8559b43f-b9d0-4fed-90a0-c01922032b21-bundle\") pod \"8559b43f-b9d0-4fed-90a0-c01922032b21\" (UID: \"8559b43f-b9d0-4fed-90a0-c01922032b21\") " Nov 02 22:53:51.961581 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.961534 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8559b43f-b9d0-4fed-90a0-c01922032b21-util\" (UniqueName: \"kubernetes.io/empty-dir/8559b43f-b9d0-4fed-90a0-c01922032b21-util\") pod \"8559b43f-b9d0-4fed-90a0-c01922032b21\" (UID: \"8559b43f-b9d0-4fed-90a0-c01922032b21\") " Nov 02 22:53:51.961636 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.961616 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c416919-9bad-4354-a369-433901bc43a7-bundle" pod "0c416919-9bad-4354-a369-433901bc43a7" (UID: "0c416919-9bad-4354-a369-433901bc43a7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:51.961881 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.961853 5727 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c416919-9bad-4354-a369-433901bc43a7-bundle\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:51.962540 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.962514 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8559b43f-b9d0-4fed-90a0-c01922032b21-bundle" pod "8559b43f-b9d0-4fed-90a0-c01922032b21" (UID: "8559b43f-b9d0-4fed-90a0-c01922032b21"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:51.963542 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.963521 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8559b43f-b9d0-4fed-90a0-c01922032b21-kube-api-access-bj58t" pod "8559b43f-b9d0-4fed-90a0-c01922032b21" (UID: "8559b43f-b9d0-4fed-90a0-c01922032b21"). InnerVolumeSpecName "kube-api-access-bj58t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:53:51.963826 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.963806 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c416919-9bad-4354-a369-433901bc43a7-kube-api-access-lq64r" pod "0c416919-9bad-4354-a369-433901bc43a7" (UID: "0c416919-9bad-4354-a369-433901bc43a7"). InnerVolumeSpecName "kube-api-access-lq64r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:53:51.967574 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.967552 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c416919-9bad-4354-a369-433901bc43a7-util" pod "0c416919-9bad-4354-a369-433901bc43a7" (UID: "0c416919-9bad-4354-a369-433901bc43a7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:51.968264 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:51.968244 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8559b43f-b9d0-4fed-90a0-c01922032b21-util" pod "8559b43f-b9d0-4fed-90a0-c01922032b21" (UID: "8559b43f-b9d0-4fed-90a0-c01922032b21"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:52.036938 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:52.036914 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-pd79m" Nov 02 22:53:52.062992 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:52.062962 5727 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8559b43f-b9d0-4fed-90a0-c01922032b21-bundle\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:52.063097 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:52.062996 5727 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8559b43f-b9d0-4fed-90a0-c01922032b21-util\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:52.063097 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:52.063011 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj58t\" (UniqueName: \"kubernetes.io/projected/8559b43f-b9d0-4fed-90a0-c01922032b21-kube-api-access-bj58t\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:52.063097 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:52.063026 5727 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c416919-9bad-4354-a369-433901bc43a7-util\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:52.063097 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:52.063039 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq64r\" (UniqueName: \"kubernetes.io/projected/0c416919-9bad-4354-a369-433901bc43a7-kube-api-access-lq64r\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:52.571437 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:52.571413 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-pd79m"] Nov 02 22:53:52.572216 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:53:52.572192 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ab5f727_4be4_466f_8cbc_10874091208d.slice/crio-b4427ba5a0c0b5a1228a04c5b1dbad0930f2f298da8911167a8fcef3313ae919 WatchSource:0}: Error finding container b4427ba5a0c0b5a1228a04c5b1dbad0930f2f298da8911167a8fcef3313ae919: Status 404 returned error can't find the container with id b4427ba5a0c0b5a1228a04c5b1dbad0930f2f298da8911167a8fcef3313ae919 Nov 02 22:53:52.573925 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:52.573909 5727 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:53:52.711577 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:52.711543 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-pd79m" event={"ID":"2ab5f727-4be4-466f-8cbc-10874091208d","Type":"ContainerStarted","Data":"b4427ba5a0c0b5a1228a04c5b1dbad0930f2f298da8911167a8fcef3313ae919"} Nov 02 22:53:52.713575 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:52.713541 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qgns6" event={"ID":"0c416919-9bad-4354-a369-433901bc43a7","Type":"ContainerDied","Data":"3b983f1c9a3f840d1a08704a9cabd0b67caa27242a643fe7b8a459eb54d74980"} Nov 02 22:53:52.713718 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:52.713594 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b983f1c9a3f840d1a08704a9cabd0b67caa27242a643fe7b8a459eb54d74980" Nov 02 22:53:52.713718 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:52.713551 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qgns6" Nov 02 22:53:52.715579 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:52.715556 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvtmn" Nov 02 22:53:52.715579 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:52.715560 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvtmn" event={"ID":"8559b43f-b9d0-4fed-90a0-c01922032b21","Type":"ContainerDied","Data":"2353a2266bc999d5d20a586f73b1c8925c643eab5ee5040d7e4dc3122104c1f4"} Nov 02 22:53:52.715791 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:52.715601 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2353a2266bc999d5d20a586f73b1c8925c643eab5ee5040d7e4dc3122104c1f4" Nov 02 22:53:52.848935 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:52.848914 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7ngb8" Nov 02 22:53:52.869321 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:52.869301 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/495baab4-1e67-4202-8cb1-ab002c5dd19a-util\" (UniqueName: \"kubernetes.io/empty-dir/495baab4-1e67-4202-8cb1-ab002c5dd19a-util\") pod \"495baab4-1e67-4202-8cb1-ab002c5dd19a\" (UID: \"495baab4-1e67-4202-8cb1-ab002c5dd19a\") " Nov 02 22:53:52.869410 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:52.869362 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/495baab4-1e67-4202-8cb1-ab002c5dd19a-kube-api-access-qvt5n\" (UniqueName: \"kubernetes.io/projected/495baab4-1e67-4202-8cb1-ab002c5dd19a-kube-api-access-qvt5n\") pod \"495baab4-1e67-4202-8cb1-ab002c5dd19a\" (UID: \"495baab4-1e67-4202-8cb1-ab002c5dd19a\") " Nov 02 22:53:52.869448 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:52.869418 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/495baab4-1e67-4202-8cb1-ab002c5dd19a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/495baab4-1e67-4202-8cb1-ab002c5dd19a-bundle\") pod \"495baab4-1e67-4202-8cb1-ab002c5dd19a\" (UID: \"495baab4-1e67-4202-8cb1-ab002c5dd19a\") " Nov 02 22:53:52.870071 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:52.870046 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495baab4-1e67-4202-8cb1-ab002c5dd19a-bundle" pod "495baab4-1e67-4202-8cb1-ab002c5dd19a" (UID: "495baab4-1e67-4202-8cb1-ab002c5dd19a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:52.871450 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:52.871422 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/495baab4-1e67-4202-8cb1-ab002c5dd19a-kube-api-access-qvt5n" pod "495baab4-1e67-4202-8cb1-ab002c5dd19a" (UID: "495baab4-1e67-4202-8cb1-ab002c5dd19a"). InnerVolumeSpecName "kube-api-access-qvt5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:53:52.874462 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:52.874442 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495baab4-1e67-4202-8cb1-ab002c5dd19a-util" pod "495baab4-1e67-4202-8cb1-ab002c5dd19a" (UID: "495baab4-1e67-4202-8cb1-ab002c5dd19a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:52.970575 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:52.970546 5727 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/495baab4-1e67-4202-8cb1-ab002c5dd19a-bundle\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:52.970575 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:52.970574 5727 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/495baab4-1e67-4202-8cb1-ab002c5dd19a-util\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:52.970733 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:52.970584 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvt5n\" (UniqueName: \"kubernetes.io/projected/495baab4-1e67-4202-8cb1-ab002c5dd19a-kube-api-access-qvt5n\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:53.723677 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:53.723639 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7ngb8" event={"ID":"495baab4-1e67-4202-8cb1-ab002c5dd19a","Type":"ContainerDied","Data":"a781c56dedf1b1fe65ed2077b9951a71c7312df01f0c26618620f49f3c51303a"} Nov 02 22:53:53.724174 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:53.723718 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a781c56dedf1b1fe65ed2077b9951a71c7312df01f0c26618620f49f3c51303a" Nov 02 22:53:53.724174 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:53.723649 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7ngb8" Nov 02 22:53:56.741185 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:56.741087 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-pd79m" event={"ID":"2ab5f727-4be4-466f-8cbc-10874091208d","Type":"ContainerStarted","Data":"bf7d83ee6def3684bb1f3e0bb1ed0f99e76c2e25b7c343740a717e254eaf3c05"} Nov 02 22:53:57.746767 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:57.746731 5727 generic.go:356] "Generic (PLEG): container finished" podID="2ab5f727-4be4-466f-8cbc-10874091208d" containerID="bf7d83ee6def3684bb1f3e0bb1ed0f99e76c2e25b7c343740a717e254eaf3c05" exitCode=0 Nov 02 22:53:57.747163 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:57.746820 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-pd79m" event={"ID":"2ab5f727-4be4-466f-8cbc-10874091208d","Type":"ContainerDied","Data":"bf7d83ee6def3684bb1f3e0bb1ed0f99e76c2e25b7c343740a717e254eaf3c05"} Nov 02 22:53:57.844204 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:53:57.844166 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 22:53:58.754362 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:58.754266 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-pd79m" event={"ID":"2ab5f727-4be4-466f-8cbc-10874091208d","Type":"ContainerStarted","Data":"22772c3f180c709355e884c330aa80416eacc63524fe2e7fa85a098ef9cc82c9"} Nov 02 22:53:58.754807 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:58.754587 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-pd79m" Nov 02 22:53:58.900807 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:53:58.900754 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-pd79m" podStartSLOduration=2.937408346 podStartE2EDuration="7.900741162s" totalImagesPullingTime="4.484545255s" totalInitContainerRuntime="478.787561ms" isStatefulPod=false podCreationTimestamp="2026-11-02 22:53:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:58.898121826 +0000 UTC m=+325.136619520" watchObservedRunningTime="2026-11-02 22:53:58.900741162 +0000 UTC m=+325.139238889" Nov 02 22:54:01.754749 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:01.754718 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-v99d8"] Nov 02 22:54:01.755118 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:01.755045 5727 cpu_manager.go:414] "containerMap: removing container" podUID="495baab4-1e67-4202-8cb1-ab002c5dd19a" containerName="util" Nov 02 22:54:01.755118 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:01.755056 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="495baab4-1e67-4202-8cb1-ab002c5dd19a" containerName="util" Nov 02 22:54:01.755118 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:01.755067 5727 cpu_manager.go:414] "containerMap: removing container" podUID="8559b43f-b9d0-4fed-90a0-c01922032b21" containerName="util" Nov 02 22:54:01.755118 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:01.755072 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8559b43f-b9d0-4fed-90a0-c01922032b21" containerName="util" Nov 02 22:54:01.755118 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:01.755080 5727 cpu_manager.go:414] "containerMap: removing container" podUID="495baab4-1e67-4202-8cb1-ab002c5dd19a" containerName="extract" Nov 02 22:54:01.755118 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:01.755085 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="495baab4-1e67-4202-8cb1-ab002c5dd19a" containerName="extract" Nov 02 22:54:01.755118 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:01.755093 5727 cpu_manager.go:414] "containerMap: removing container" podUID="0c416919-9bad-4354-a369-433901bc43a7" containerName="pull" Nov 02 22:54:01.755118 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:01.755098 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c416919-9bad-4354-a369-433901bc43a7" containerName="pull" Nov 02 22:54:01.755118 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:01.755114 5727 cpu_manager.go:414] "containerMap: removing container" podUID="8559b43f-b9d0-4fed-90a0-c01922032b21" containerName="pull" Nov 02 22:54:01.755118 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:01.755119 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8559b43f-b9d0-4fed-90a0-c01922032b21" containerName="pull" Nov 02 22:54:01.755118 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:01.755137 5727 cpu_manager.go:414] "containerMap: removing container" podUID="495baab4-1e67-4202-8cb1-ab002c5dd19a" containerName="pull" Nov 02 22:54:01.755465 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:01.755150 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="495baab4-1e67-4202-8cb1-ab002c5dd19a" containerName="pull" Nov 02 22:54:01.755465 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:01.755159 5727 cpu_manager.go:414] "containerMap: removing container" podUID="8559b43f-b9d0-4fed-90a0-c01922032b21" containerName="extract" Nov 02 22:54:01.755465 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:01.755164 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8559b43f-b9d0-4fed-90a0-c01922032b21" containerName="extract" Nov 02 22:54:01.755465 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:01.755177 5727 cpu_manager.go:414] "containerMap: removing container" podUID="0c416919-9bad-4354-a369-433901bc43a7" containerName="util" Nov 02 22:54:01.755465 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:01.755181 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c416919-9bad-4354-a369-433901bc43a7" containerName="util" Nov 02 22:54:01.755465 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:01.755188 5727 cpu_manager.go:414] "containerMap: removing container" podUID="0c416919-9bad-4354-a369-433901bc43a7" containerName="extract" Nov 02 22:54:01.755465 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:01.755193 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c416919-9bad-4354-a369-433901bc43a7" containerName="extract" Nov 02 22:54:01.755465 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:01.755243 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="8559b43f-b9d0-4fed-90a0-c01922032b21" containerName="extract" Nov 02 22:54:01.755465 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:01.755252 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c416919-9bad-4354-a369-433901bc43a7" containerName="extract" Nov 02 22:54:01.755465 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:01.755259 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="495baab4-1e67-4202-8cb1-ab002c5dd19a" containerName="extract" Nov 02 22:54:01.757450 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:01.757434 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-v99d8" Nov 02 22:54:01.813766 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:01.813737 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-v99d8"] Nov 02 22:54:01.842068 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:01.842043 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3402de82-da8f-4df9-b696-13cdebfb926b-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-v99d8\" (UID: \"3402de82-da8f-4df9-b696-13cdebfb926b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-v99d8" Nov 02 22:54:01.842202 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:01.842074 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pcbm\" (UniqueName: \"kubernetes.io/projected/3402de82-da8f-4df9-b696-13cdebfb926b-kube-api-access-9pcbm\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-v99d8\" (UID: \"3402de82-da8f-4df9-b696-13cdebfb926b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-v99d8" Nov 02 22:54:01.861317 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:01.861295 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 02 22:54:01.861438 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:01.861292 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-2vwhg\"" Nov 02 22:54:01.861492 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:01.861461 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:54:01.943280 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:01.943255 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3402de82-da8f-4df9-b696-13cdebfb926b-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-v99d8\" (UID: \"3402de82-da8f-4df9-b696-13cdebfb926b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-v99d8" Nov 02 22:54:01.943280 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:01.943286 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pcbm\" (UniqueName: \"kubernetes.io/projected/3402de82-da8f-4df9-b696-13cdebfb926b-kube-api-access-9pcbm\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-v99d8\" (UID: \"3402de82-da8f-4df9-b696-13cdebfb926b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-v99d8" Nov 02 22:54:01.943624 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:01.943603 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3402de82-da8f-4df9-b696-13cdebfb926b-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-v99d8\" (UID: \"3402de82-da8f-4df9-b696-13cdebfb926b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-v99d8" Nov 02 22:54:02.007951 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:02.007892 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pcbm\" (UniqueName: \"kubernetes.io/projected/3402de82-da8f-4df9-b696-13cdebfb926b-kube-api-access-9pcbm\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-v99d8\" (UID: \"3402de82-da8f-4df9-b696-13cdebfb926b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-v99d8" Nov 02 22:54:02.068969 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:02.068937 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-v99d8" Nov 02 22:54:02.497166 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:02.497122 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-v99d8"] Nov 02 22:54:02.499522 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:54:02.499490 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3402de82_da8f_4df9_b696_13cdebfb926b.slice/crio-4adc27e6b7401cbf68c64315144b3af41e075c4e907fb7e62d110ff911b873bb WatchSource:0}: Error finding container 4adc27e6b7401cbf68c64315144b3af41e075c4e907fb7e62d110ff911b873bb: Status 404 returned error can't find the container with id 4adc27e6b7401cbf68c64315144b3af41e075c4e907fb7e62d110ff911b873bb Nov 02 22:54:02.774884 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:02.774809 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-v99d8" event={"ID":"3402de82-da8f-4df9-b696-13cdebfb926b","Type":"ContainerStarted","Data":"4adc27e6b7401cbf68c64315144b3af41e075c4e907fb7e62d110ff911b873bb"} Nov 02 22:54:09.762079 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:09.762050 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-pd79m" Nov 02 22:54:15.842666 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:15.842625 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-v99d8" event={"ID":"3402de82-da8f-4df9-b696-13cdebfb926b","Type":"ContainerStarted","Data":"775aebc6781da2a48ef905809dbd8bba9d12fab5ad62dfbc10d185eb0a170702"} Nov 02 22:54:16.073708 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:16.073657 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-v99d8" podStartSLOduration=2.460502915 podStartE2EDuration="15.073643069s" totalImagesPullingTime="12.613140154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:54:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:54:16.07241215 +0000 UTC m=+342.310909826" watchObservedRunningTime="2026-11-02 22:54:16.073643069 +0000 UTC m=+342.312140731" Nov 02 22:54:17.729005 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:17.728969 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-8kcd5"] Nov 02 22:54:17.732545 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:17.732528 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8kcd5" Nov 02 22:54:17.772984 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:17.772955 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4af923f3-9d59-4615-862e-dcf89208c73e-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-8kcd5\" (UID: \"4af923f3-9d59-4615-862e-dcf89208c73e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8kcd5" Nov 02 22:54:17.773093 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:17.773008 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s97qf\" (UniqueName: \"kubernetes.io/projected/4af923f3-9d59-4615-862e-dcf89208c73e-kube-api-access-s97qf\") pod \"cert-manager-webhook-5dbbbf75cc-8kcd5\" (UID: \"4af923f3-9d59-4615-862e-dcf89208c73e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8kcd5" Nov 02 22:54:17.841560 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:17.841540 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 02 22:54:17.841675 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:17.841573 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 02 22:54:17.841675 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:17.841620 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-vgfqx\"" Nov 02 22:54:17.873985 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:17.873966 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s97qf\" (UniqueName: \"kubernetes.io/projected/4af923f3-9d59-4615-862e-dcf89208c73e-kube-api-access-s97qf\") pod \"cert-manager-webhook-5dbbbf75cc-8kcd5\" (UID: \"4af923f3-9d59-4615-862e-dcf89208c73e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8kcd5" Nov 02 22:54:17.874104 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:17.874010 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4af923f3-9d59-4615-862e-dcf89208c73e-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-8kcd5\" (UID: \"4af923f3-9d59-4615-862e-dcf89208c73e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8kcd5" Nov 02 22:54:17.887884 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:17.887862 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-8kcd5"] Nov 02 22:54:17.947693 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:17.947670 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4af923f3-9d59-4615-862e-dcf89208c73e-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-8kcd5\" (UID: \"4af923f3-9d59-4615-862e-dcf89208c73e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8kcd5" Nov 02 22:54:17.966537 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:17.966507 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s97qf\" (UniqueName: \"kubernetes.io/projected/4af923f3-9d59-4615-862e-dcf89208c73e-kube-api-access-s97qf\") pod \"cert-manager-webhook-5dbbbf75cc-8kcd5\" (UID: \"4af923f3-9d59-4615-862e-dcf89208c73e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8kcd5" Nov 02 22:54:18.054996 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:18.054931 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8kcd5" Nov 02 22:54:18.427215 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:18.427194 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-8kcd5"] Nov 02 22:54:18.429359 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:54:18.429326 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4af923f3_9d59_4615_862e_dcf89208c73e.slice/crio-a3e1df76012d01d7c48046bd9c89d9cc5bea6fc29732d07202d45b839b2295d8 WatchSource:0}: Error finding container a3e1df76012d01d7c48046bd9c89d9cc5bea6fc29732d07202d45b839b2295d8: Status 404 returned error can't find the container with id a3e1df76012d01d7c48046bd9c89d9cc5bea6fc29732d07202d45b839b2295d8 Nov 02 22:54:18.857754 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:18.857709 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8kcd5" event={"ID":"4af923f3-9d59-4615-862e-dcf89208c73e","Type":"ContainerStarted","Data":"a3e1df76012d01d7c48046bd9c89d9cc5bea6fc29732d07202d45b839b2295d8"} Nov 02 22:54:23.510505 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:23.510476 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-ptd66"] Nov 02 22:54:23.515199 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:23.515184 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ptd66" Nov 02 22:54:23.571031 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:23.571002 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 22:54:23.571031 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:23.571015 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-ghmtm\"" Nov 02 22:54:23.571031 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:23.571034 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 22:54:23.620002 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:23.619962 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvhtn\" (UniqueName: \"kubernetes.io/projected/de8616c3-ba9c-4b99-813e-ef4c190fc4c7-kube-api-access-kvhtn\") pod \"authorino-operator-7b7bb4bc77-ptd66\" (UID: \"de8616c3-ba9c-4b99-813e-ef4c190fc4c7\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-ptd66" Nov 02 22:54:23.623301 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:23.623276 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-ptd66"] Nov 02 22:54:23.720579 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:23.720547 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvhtn\" (UniqueName: \"kubernetes.io/projected/de8616c3-ba9c-4b99-813e-ef4c190fc4c7-kube-api-access-kvhtn\") pod \"authorino-operator-7b7bb4bc77-ptd66\" (UID: \"de8616c3-ba9c-4b99-813e-ef4c190fc4c7\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-ptd66" Nov 02 22:54:23.781075 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:23.781005 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvhtn\" (UniqueName: \"kubernetes.io/projected/de8616c3-ba9c-4b99-813e-ef4c190fc4c7-kube-api-access-kvhtn\") pod \"authorino-operator-7b7bb4bc77-ptd66\" (UID: \"de8616c3-ba9c-4b99-813e-ef4c190fc4c7\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-ptd66" Nov 02 22:54:23.828694 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:23.828665 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ptd66" Nov 02 22:54:24.206753 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:24.206730 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-ptd66"] Nov 02 22:54:24.209568 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:54:24.209540 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde8616c3_ba9c_4b99_813e_ef4c190fc4c7.slice/crio-685697cac2dc72ffe97fe91919b88f51a05d12ddecbffa00c8aa72f2d98e08eb WatchSource:0}: Error finding container 685697cac2dc72ffe97fe91919b88f51a05d12ddecbffa00c8aa72f2d98e08eb: Status 404 returned error can't find the container with id 685697cac2dc72ffe97fe91919b88f51a05d12ddecbffa00c8aa72f2d98e08eb Nov 02 22:54:24.890120 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:24.890082 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ptd66" event={"ID":"de8616c3-ba9c-4b99-813e-ef4c190fc4c7","Type":"ContainerStarted","Data":"685697cac2dc72ffe97fe91919b88f51a05d12ddecbffa00c8aa72f2d98e08eb"} Nov 02 22:54:25.730918 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:25.730883 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-x6h9v"] Nov 02 22:54:25.735372 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:25.735356 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x6h9v" Nov 02 22:54:25.837251 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:25.837212 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6j8m\" (UniqueName: \"kubernetes.io/projected/349fa078-8c82-4c07-a596-900fbb745d7f-kube-api-access-r6j8m\") pod \"limitador-operator-controller-manager-5778bb4449-x6h9v\" (UID: \"349fa078-8c82-4c07-a596-900fbb745d7f\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x6h9v" Nov 02 22:54:25.839847 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:25.839820 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-jbjfp\"" Nov 02 22:54:25.842229 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:25.842202 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-x6h9v"] Nov 02 22:54:25.938225 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:25.938182 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6j8m\" (UniqueName: \"kubernetes.io/projected/349fa078-8c82-4c07-a596-900fbb745d7f-kube-api-access-r6j8m\") pod \"limitador-operator-controller-manager-5778bb4449-x6h9v\" (UID: \"349fa078-8c82-4c07-a596-900fbb745d7f\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x6h9v" Nov 02 22:54:26.000727 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:26.000638 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6j8m\" (UniqueName: \"kubernetes.io/projected/349fa078-8c82-4c07-a596-900fbb745d7f-kube-api-access-r6j8m\") pod \"limitador-operator-controller-manager-5778bb4449-x6h9v\" (UID: \"349fa078-8c82-4c07-a596-900fbb745d7f\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x6h9v" Nov 02 22:54:26.048448 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:26.048225 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x6h9v" Nov 02 22:54:26.454488 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:26.454451 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-x6h9v"] Nov 02 22:54:26.458309 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:54:26.458278 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod349fa078_8c82_4c07_a596_900fbb745d7f.slice/crio-d1b92d0e24d3851f9f56fef387aad36dfe098a043a0d740aa2c8b4114e306d6c WatchSource:0}: Error finding container d1b92d0e24d3851f9f56fef387aad36dfe098a043a0d740aa2c8b4114e306d6c: Status 404 returned error can't find the container with id d1b92d0e24d3851f9f56fef387aad36dfe098a043a0d740aa2c8b4114e306d6c Nov 02 22:54:26.902827 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:26.902782 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x6h9v" event={"ID":"349fa078-8c82-4c07-a596-900fbb745d7f","Type":"ContainerStarted","Data":"d1b92d0e24d3851f9f56fef387aad36dfe098a043a0d740aa2c8b4114e306d6c"} Nov 02 22:54:27.844601 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:54:27.844560 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 22:54:28.917643 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:28.917039 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ptd66" event={"ID":"de8616c3-ba9c-4b99-813e-ef4c190fc4c7","Type":"ContainerStarted","Data":"5d5dd472c05b60a34531076f71b156c4d8b2fea560de458155c0e8e4cec8fa5f"} Nov 02 22:54:28.917643 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:28.917340 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ptd66" Nov 02 22:54:29.923495 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:29.923458 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x6h9v" event={"ID":"349fa078-8c82-4c07-a596-900fbb745d7f","Type":"ContainerStarted","Data":"2aa5408d4b51030c955b6fb67b3e6371c269658815b104b5c386157e6e44f36f"} Nov 02 22:54:29.923991 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:29.923754 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x6h9v" Nov 02 22:54:30.051456 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:30.051396 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ptd66" podStartSLOduration=3.202708098 podStartE2EDuration="7.051377569s" totalImagesPullingTime="3.848669471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:54:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:54:29.042830017 +0000 UTC m=+355.281327699" watchObservedRunningTime="2026-11-02 22:54:30.051377569 +0000 UTC m=+356.289875231" Nov 02 22:54:30.052685 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:30.052652 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x6h9v" podStartSLOduration=2.525648764 podStartE2EDuration="5.052642549s" totalImagesPullingTime="2.526993785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:54:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:54:30.050198571 +0000 UTC m=+356.288696312" watchObservedRunningTime="2026-11-02 22:54:30.052642549 +0000 UTC m=+356.291140209" Nov 02 22:54:30.230632 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:30.230549 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gxmgl"] Nov 02 22:54:30.233842 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:30.233822 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gxmgl" Nov 02 22:54:30.296746 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:30.296716 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6vkc2\"" Nov 02 22:54:30.367486 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:30.360203 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gxmgl"] Nov 02 22:54:30.381461 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:30.381430 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd31caf7-718a-4852-9aeb-6dc5b8bd05c2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-gxmgl\" (UID: \"fd31caf7-718a-4852-9aeb-6dc5b8bd05c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gxmgl" Nov 02 22:54:30.381586 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:30.381475 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbfb4\" (UniqueName: \"kubernetes.io/projected/fd31caf7-718a-4852-9aeb-6dc5b8bd05c2-kube-api-access-lbfb4\") pod \"kuadrant-operator-controller-manager-678d87766f-gxmgl\" (UID: \"fd31caf7-718a-4852-9aeb-6dc5b8bd05c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gxmgl" Nov 02 22:54:30.482163 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:30.482048 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbfb4\" (UniqueName: \"kubernetes.io/projected/fd31caf7-718a-4852-9aeb-6dc5b8bd05c2-kube-api-access-lbfb4\") pod \"kuadrant-operator-controller-manager-678d87766f-gxmgl\" (UID: \"fd31caf7-718a-4852-9aeb-6dc5b8bd05c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gxmgl" Nov 02 22:54:30.482323 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:30.482166 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd31caf7-718a-4852-9aeb-6dc5b8bd05c2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-gxmgl\" (UID: \"fd31caf7-718a-4852-9aeb-6dc5b8bd05c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gxmgl" Nov 02 22:54:30.482545 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:30.482526 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd31caf7-718a-4852-9aeb-6dc5b8bd05c2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-gxmgl\" (UID: \"fd31caf7-718a-4852-9aeb-6dc5b8bd05c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gxmgl" Nov 02 22:54:30.551364 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:30.551330 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbfb4\" (UniqueName: \"kubernetes.io/projected/fd31caf7-718a-4852-9aeb-6dc5b8bd05c2-kube-api-access-lbfb4\") pod \"kuadrant-operator-controller-manager-678d87766f-gxmgl\" (UID: \"fd31caf7-718a-4852-9aeb-6dc5b8bd05c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gxmgl" Nov 02 22:54:30.846900 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:30.846876 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gxmgl" Nov 02 22:54:30.933048 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:30.933009 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8kcd5" event={"ID":"4af923f3-9d59-4615-862e-dcf89208c73e","Type":"ContainerStarted","Data":"8c6e9f71e9e2a4f14b942d732fa31331b1b2ba7bbc0e2abcc058106d9feed29e"} Nov 02 22:54:30.933454 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:30.933069 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8kcd5" Nov 02 22:54:31.058353 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:31.058300 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8kcd5" podStartSLOduration=1.664425426 podStartE2EDuration="14.058285572s" totalImagesPullingTime="12.393860146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:54:31.056501145 +0000 UTC m=+357.294998809" watchObservedRunningTime="2026-11-02 22:54:31.058285572 +0000 UTC m=+357.296783233" Nov 02 22:54:31.349953 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:31.349931 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gxmgl"] Nov 02 22:54:31.355243 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:54:31.355214 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd31caf7_718a_4852_9aeb_6dc5b8bd05c2.slice/crio-6c49c18108b4cf7e8870a2ecbd92a96a4f1f9195980a66e9155f036d50b26d57 WatchSource:0}: Error finding container 6c49c18108b4cf7e8870a2ecbd92a96a4f1f9195980a66e9155f036d50b26d57: Status 404 returned error can't find the container with id 6c49c18108b4cf7e8870a2ecbd92a96a4f1f9195980a66e9155f036d50b26d57 Nov 02 22:54:31.938482 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:31.938440 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gxmgl" event={"ID":"fd31caf7-718a-4852-9aeb-6dc5b8bd05c2","Type":"ContainerStarted","Data":"6c49c18108b4cf7e8870a2ecbd92a96a4f1f9195980a66e9155f036d50b26d57"} Nov 02 22:54:32.114989 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:32.114957 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-cp9j8"] Nov 02 22:54:32.118845 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:32.118830 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cp9j8" Nov 02 22:54:32.176383 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:32.176353 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 02 22:54:32.196970 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:32.196910 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j96vp\" (UniqueName: \"kubernetes.io/projected/e5198e9e-a261-4491-93da-4fad7b359312-kube-api-access-j96vp\") pod \"dns-operator-controller-manager-8557cbdb84-cp9j8\" (UID: \"e5198e9e-a261-4491-93da-4fad7b359312\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cp9j8" Nov 02 22:54:32.225728 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:32.225698 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-trtzc\"" Nov 02 22:54:32.227533 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:32.227489 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-cp9j8"] Nov 02 22:54:32.297375 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:32.297343 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j96vp\" (UniqueName: \"kubernetes.io/projected/e5198e9e-a261-4491-93da-4fad7b359312-kube-api-access-j96vp\") pod \"dns-operator-controller-manager-8557cbdb84-cp9j8\" (UID: \"e5198e9e-a261-4491-93da-4fad7b359312\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cp9j8" Nov 02 22:54:32.360955 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:32.360922 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j96vp\" (UniqueName: \"kubernetes.io/projected/e5198e9e-a261-4491-93da-4fad7b359312-kube-api-access-j96vp\") pod \"dns-operator-controller-manager-8557cbdb84-cp9j8\" (UID: \"e5198e9e-a261-4491-93da-4fad7b359312\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cp9j8" Nov 02 22:54:32.431427 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:32.431395 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cp9j8" Nov 02 22:54:32.808694 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:32.808661 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-cp9j8"] Nov 02 22:54:32.810894 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:54:32.810867 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5198e9e_a261_4491_93da_4fad7b359312.slice/crio-9a45f01d6fe8bdf12b5fff6ee0cc3fbaebb565b2619c1e256ab6bd1eaa2a3a19 WatchSource:0}: Error finding container 9a45f01d6fe8bdf12b5fff6ee0cc3fbaebb565b2619c1e256ab6bd1eaa2a3a19: Status 404 returned error can't find the container with id 9a45f01d6fe8bdf12b5fff6ee0cc3fbaebb565b2619c1e256ab6bd1eaa2a3a19 Nov 02 22:54:32.947422 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:32.947386 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cp9j8" event={"ID":"e5198e9e-a261-4491-93da-4fad7b359312","Type":"ContainerStarted","Data":"9a45f01d6fe8bdf12b5fff6ee0cc3fbaebb565b2619c1e256ab6bd1eaa2a3a19"} Nov 02 22:54:36.040259 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:36.040233 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 02 22:54:36.940558 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:36.940524 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8kcd5" Nov 02 22:54:36.973882 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:36.973845 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gxmgl" event={"ID":"fd31caf7-718a-4852-9aeb-6dc5b8bd05c2","Type":"ContainerStarted","Data":"638349bf9cea9a72a719d91b644c2776a656762f9e1ab6603d0ca3fac70a0903"} Nov 02 22:54:36.974051 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:36.973908 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gxmgl" Nov 02 22:54:36.975626 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:36.975600 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cp9j8" event={"ID":"e5198e9e-a261-4491-93da-4fad7b359312","Type":"ContainerStarted","Data":"aa395c95c7893dd3e0395b2f533678fcd4e3abb0013a9c3276fe23be045b8f57"} Nov 02 22:54:36.975754 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:36.975650 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cp9j8" Nov 02 22:54:37.178084 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:37.178028 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cp9j8" podStartSLOduration=2.002996561 podStartE2EDuration="5.178014269s" totalImagesPullingTime="3.175017708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:54:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:54:37.176317242 +0000 UTC m=+363.414814902" watchObservedRunningTime="2026-11-02 22:54:37.178014269 +0000 UTC m=+363.416511929" Nov 02 22:54:37.302485 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:37.302388 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gxmgl" podStartSLOduration=2.238555085 podStartE2EDuration="7.302374293s" totalImagesPullingTime="5.063819208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:54:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:54:37.301687658 +0000 UTC m=+363.540185340" watchObservedRunningTime="2026-11-02 22:54:37.302374293 +0000 UTC m=+363.540871954" Nov 02 22:54:39.925806 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:39.925780 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ptd66" Nov 02 22:54:40.712078 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:40.712047 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr"] Nov 02 22:54:40.715741 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:40.715713 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:40.777197 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:40.777168 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-276tc\"" Nov 02 22:54:40.861707 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:40.861669 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr"] Nov 02 22:54:40.873793 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:40.873767 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/856fc499-f20f-41cd-8baf-ed58d33c45c5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr\" (UID: \"856fc499-f20f-41cd-8baf-ed58d33c45c5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:40.873932 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:40.873813 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/856fc499-f20f-41cd-8baf-ed58d33c45c5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr\" (UID: \"856fc499-f20f-41cd-8baf-ed58d33c45c5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:40.873932 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:40.873833 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmzwn\" (UniqueName: \"kubernetes.io/projected/856fc499-f20f-41cd-8baf-ed58d33c45c5-kube-api-access-nmzwn\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr\" (UID: \"856fc499-f20f-41cd-8baf-ed58d33c45c5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:40.874017 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:40.873934 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/856fc499-f20f-41cd-8baf-ed58d33c45c5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr\" (UID: \"856fc499-f20f-41cd-8baf-ed58d33c45c5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:40.874060 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:40.874046 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/856fc499-f20f-41cd-8baf-ed58d33c45c5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr\" (UID: \"856fc499-f20f-41cd-8baf-ed58d33c45c5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:40.874095 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:40.874075 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/856fc499-f20f-41cd-8baf-ed58d33c45c5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr\" (UID: \"856fc499-f20f-41cd-8baf-ed58d33c45c5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:40.874148 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:40.874111 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/856fc499-f20f-41cd-8baf-ed58d33c45c5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr\" (UID: \"856fc499-f20f-41cd-8baf-ed58d33c45c5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:40.874203 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:40.874182 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/856fc499-f20f-41cd-8baf-ed58d33c45c5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr\" (UID: \"856fc499-f20f-41cd-8baf-ed58d33c45c5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:40.874244 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:40.874229 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/856fc499-f20f-41cd-8baf-ed58d33c45c5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr\" (UID: \"856fc499-f20f-41cd-8baf-ed58d33c45c5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:40.935457 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:40.935431 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x6h9v" Nov 02 22:54:40.974821 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:40.974734 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/856fc499-f20f-41cd-8baf-ed58d33c45c5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr\" (UID: \"856fc499-f20f-41cd-8baf-ed58d33c45c5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:40.974821 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:40.974772 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/856fc499-f20f-41cd-8baf-ed58d33c45c5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr\" (UID: \"856fc499-f20f-41cd-8baf-ed58d33c45c5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:40.974821 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:40.974793 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/856fc499-f20f-41cd-8baf-ed58d33c45c5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr\" (UID: \"856fc499-f20f-41cd-8baf-ed58d33c45c5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:40.975056 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:40.974919 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/856fc499-f20f-41cd-8baf-ed58d33c45c5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr\" (UID: \"856fc499-f20f-41cd-8baf-ed58d33c45c5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:40.975056 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:40.974977 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/856fc499-f20f-41cd-8baf-ed58d33c45c5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr\" (UID: \"856fc499-f20f-41cd-8baf-ed58d33c45c5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:40.975056 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:40.975020 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/856fc499-f20f-41cd-8baf-ed58d33c45c5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr\" (UID: \"856fc499-f20f-41cd-8baf-ed58d33c45c5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:40.975227 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:40.975075 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/856fc499-f20f-41cd-8baf-ed58d33c45c5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr\" (UID: \"856fc499-f20f-41cd-8baf-ed58d33c45c5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:40.975227 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:40.975099 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmzwn\" (UniqueName: \"kubernetes.io/projected/856fc499-f20f-41cd-8baf-ed58d33c45c5-kube-api-access-nmzwn\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr\" (UID: \"856fc499-f20f-41cd-8baf-ed58d33c45c5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:40.975227 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:40.975159 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/856fc499-f20f-41cd-8baf-ed58d33c45c5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr\" (UID: \"856fc499-f20f-41cd-8baf-ed58d33c45c5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:40.975227 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:40.975191 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/856fc499-f20f-41cd-8baf-ed58d33c45c5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr\" (UID: \"856fc499-f20f-41cd-8baf-ed58d33c45c5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:40.975594 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:40.975571 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/856fc499-f20f-41cd-8baf-ed58d33c45c5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr\" (UID: \"856fc499-f20f-41cd-8baf-ed58d33c45c5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:40.975694 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:40.975642 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/856fc499-f20f-41cd-8baf-ed58d33c45c5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr\" (UID: \"856fc499-f20f-41cd-8baf-ed58d33c45c5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:40.975760 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:40.975743 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/856fc499-f20f-41cd-8baf-ed58d33c45c5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr\" (UID: \"856fc499-f20f-41cd-8baf-ed58d33c45c5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:40.976057 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:40.976034 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/856fc499-f20f-41cd-8baf-ed58d33c45c5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr\" (UID: \"856fc499-f20f-41cd-8baf-ed58d33c45c5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:40.977528 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:40.977502 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/856fc499-f20f-41cd-8baf-ed58d33c45c5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr\" (UID: \"856fc499-f20f-41cd-8baf-ed58d33c45c5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:40.977719 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:40.977696 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/856fc499-f20f-41cd-8baf-ed58d33c45c5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr\" (UID: \"856fc499-f20f-41cd-8baf-ed58d33c45c5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:41.036081 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:41.036051 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmzwn\" (UniqueName: \"kubernetes.io/projected/856fc499-f20f-41cd-8baf-ed58d33c45c5-kube-api-access-nmzwn\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr\" (UID: \"856fc499-f20f-41cd-8baf-ed58d33c45c5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:41.036251 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:41.036195 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/856fc499-f20f-41cd-8baf-ed58d33c45c5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr\" (UID: \"856fc499-f20f-41cd-8baf-ed58d33c45c5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:41.328731 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:41.328706 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:41.838385 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:41.838361 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr"] Nov 02 22:54:41.840477 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:54:41.840445 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856fc499_f20f_41cd_8baf_ed58d33c45c5.slice/crio-bc63399d5eff29b8f9dd826b7ff40e370a3c49c56fb2dc8cd6cd25f312da7d1a WatchSource:0}: Error finding container bc63399d5eff29b8f9dd826b7ff40e370a3c49c56fb2dc8cd6cd25f312da7d1a: Status 404 returned error can't find the container with id bc63399d5eff29b8f9dd826b7ff40e370a3c49c56fb2dc8cd6cd25f312da7d1a Nov 02 22:54:41.842741 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:41.842713 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 22:54:41.842814 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:41.842772 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 22:54:41.842814 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:41.842802 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 22:54:42.002522 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:42.002481 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" event={"ID":"856fc499-f20f-41cd-8baf-ed58d33c45c5","Type":"ContainerStarted","Data":"4cf073922e7c047b9b34ee5a80924d8e186782645fb756972ceceb93f8256688"} Nov 02 22:54:42.003007 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:42.002536 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" event={"ID":"856fc499-f20f-41cd-8baf-ed58d33c45c5","Type":"ContainerStarted","Data":"bc63399d5eff29b8f9dd826b7ff40e370a3c49c56fb2dc8cd6cd25f312da7d1a"} Nov 02 22:54:42.129353 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:42.129263 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" podStartSLOduration=2.129247509 podStartE2EDuration="2.129247509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:54:42.126353129 +0000 UTC m=+368.364850813" watchObservedRunningTime="2026-11-02 22:54:42.129247509 +0000 UTC m=+368.367745231" Nov 02 22:54:42.329570 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:42.329533 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:42.334265 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:42.334244 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:43.009022 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:43.008991 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:43.009821 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:43.009802 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr" Nov 02 22:54:43.813035 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:43.813001 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lth6t"] Nov 02 22:54:43.816597 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:43.816581 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-lth6t" Nov 02 22:54:43.881683 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:43.881644 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lth6t"] Nov 02 22:54:43.889616 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:43.889592 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dpz6v\"" Nov 02 22:54:43.899977 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:43.899952 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86chv\" (UniqueName: \"kubernetes.io/projected/dcea57b2-2614-41a6-a4a9-7d40c86732a6-kube-api-access-86chv\") pod \"authorino-57f7f8659f-lth6t\" (UID: \"dcea57b2-2614-41a6-a4a9-7d40c86732a6\") " pod="kuadrant-system/authorino-57f7f8659f-lth6t" Nov 02 22:54:44.001329 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:44.001297 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86chv\" (UniqueName: \"kubernetes.io/projected/dcea57b2-2614-41a6-a4a9-7d40c86732a6-kube-api-access-86chv\") pod \"authorino-57f7f8659f-lth6t\" (UID: \"dcea57b2-2614-41a6-a4a9-7d40c86732a6\") " pod="kuadrant-system/authorino-57f7f8659f-lth6t" Nov 02 22:54:44.061886 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:44.061841 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86chv\" (UniqueName: \"kubernetes.io/projected/dcea57b2-2614-41a6-a4a9-7d40c86732a6-kube-api-access-86chv\") pod \"authorino-57f7f8659f-lth6t\" (UID: \"dcea57b2-2614-41a6-a4a9-7d40c86732a6\") " pod="kuadrant-system/authorino-57f7f8659f-lth6t" Nov 02 22:54:44.129009 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:44.128943 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-lth6t" Nov 02 22:54:44.505430 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:44.505403 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lth6t"] Nov 02 22:54:44.507953 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:54:44.507925 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddcea57b2_2614_41a6_a4a9_7d40c86732a6.slice/crio-b26508cd53304c49900d60add9c1842325ffd8ead2fc997c1a5db92a81ba1285 WatchSource:0}: Error finding container b26508cd53304c49900d60add9c1842325ffd8ead2fc997c1a5db92a81ba1285: Status 404 returned error can't find the container with id b26508cd53304c49900d60add9c1842325ffd8ead2fc997c1a5db92a81ba1285 Nov 02 22:54:45.019668 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:45.019626 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-lth6t" event={"ID":"dcea57b2-2614-41a6-a4a9-7d40c86732a6","Type":"ContainerStarted","Data":"b26508cd53304c49900d60add9c1842325ffd8ead2fc997c1a5db92a81ba1285"} Nov 02 22:54:46.878431 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:46.878398 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr9qg"] Nov 02 22:54:46.881791 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:46.881772 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr9qg" Nov 02 22:54:46.936704 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:46.936673 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 02 22:54:46.939418 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:46.939389 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr9qg"] Nov 02 22:54:46.983402 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:46.983378 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 02 22:54:46.983617 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:46.983601 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-v4tz9\"" Nov 02 22:54:47.032474 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:47.032441 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1394c83b-9059-4e3e-928f-bde172669b09-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jr9qg\" (UID: \"1394c83b-9059-4e3e-928f-bde172669b09\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr9qg" Nov 02 22:54:47.032630 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:47.032553 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1394c83b-9059-4e3e-928f-bde172669b09-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jr9qg\" (UID: \"1394c83b-9059-4e3e-928f-bde172669b09\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr9qg" Nov 02 22:54:47.032630 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:47.032583 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbsvv\" (UniqueName: \"kubernetes.io/projected/1394c83b-9059-4e3e-928f-bde172669b09-kube-api-access-zbsvv\") pod \"kuadrant-console-plugin-66c4fb575d-jr9qg\" (UID: \"1394c83b-9059-4e3e-928f-bde172669b09\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr9qg" Nov 02 22:54:47.133943 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:47.133867 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1394c83b-9059-4e3e-928f-bde172669b09-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jr9qg\" (UID: \"1394c83b-9059-4e3e-928f-bde172669b09\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr9qg" Nov 02 22:54:47.133943 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:47.133929 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1394c83b-9059-4e3e-928f-bde172669b09-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jr9qg\" (UID: \"1394c83b-9059-4e3e-928f-bde172669b09\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr9qg" Nov 02 22:54:47.133943 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:47.133946 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbsvv\" (UniqueName: \"kubernetes.io/projected/1394c83b-9059-4e3e-928f-bde172669b09-kube-api-access-zbsvv\") pod \"kuadrant-console-plugin-66c4fb575d-jr9qg\" (UID: \"1394c83b-9059-4e3e-928f-bde172669b09\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr9qg" Nov 02 22:54:47.134601 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:47.134572 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1394c83b-9059-4e3e-928f-bde172669b09-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jr9qg\" (UID: \"1394c83b-9059-4e3e-928f-bde172669b09\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr9qg" Nov 02 22:54:47.136427 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:47.136403 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1394c83b-9059-4e3e-928f-bde172669b09-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jr9qg\" (UID: \"1394c83b-9059-4e3e-928f-bde172669b09\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr9qg" Nov 02 22:54:47.201633 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:47.201596 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbsvv\" (UniqueName: \"kubernetes.io/projected/1394c83b-9059-4e3e-928f-bde172669b09-kube-api-access-zbsvv\") pod \"kuadrant-console-plugin-66c4fb575d-jr9qg\" (UID: \"1394c83b-9059-4e3e-928f-bde172669b09\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr9qg" Nov 02 22:54:47.499569 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:47.499490 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr9qg" Nov 02 22:54:47.896051 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:47.895722 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr9qg"] Nov 02 22:54:47.896757 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:54:47.896726 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1394c83b_9059_4e3e_928f_bde172669b09.slice/crio-6bfc04a2cf0238eaea0a418b7f1fb9df1acf91bd9a60516db62c3e7f7684d586 WatchSource:0}: Error finding container 6bfc04a2cf0238eaea0a418b7f1fb9df1acf91bd9a60516db62c3e7f7684d586: Status 404 returned error can't find the container with id 6bfc04a2cf0238eaea0a418b7f1fb9df1acf91bd9a60516db62c3e7f7684d586 Nov 02 22:54:47.982764 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:47.982741 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cp9j8" Nov 02 22:54:47.982908 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:47.982818 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gxmgl" Nov 02 22:54:48.036458 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:48.036426 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-lth6t" event={"ID":"dcea57b2-2614-41a6-a4a9-7d40c86732a6","Type":"ContainerStarted","Data":"791421f61642836a8169d4d450bdf16b029f1b0461254c8031674a67d0cbda01"} Nov 02 22:54:48.037857 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:48.037831 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr9qg" event={"ID":"1394c83b-9059-4e3e-928f-bde172669b09","Type":"ContainerStarted","Data":"6bfc04a2cf0238eaea0a418b7f1fb9df1acf91bd9a60516db62c3e7f7684d586"} Nov 02 22:54:48.346741 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:48.346679 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-lth6t" podStartSLOduration=2.05833115 podStartE2EDuration="5.346665044s" totalImagesPullingTime="3.288333894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:54:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:54:48.344660975 +0000 UTC m=+374.583158656" watchObservedRunningTime="2026-11-02 22:54:48.346665044 +0000 UTC m=+374.585162706" Nov 02 22:54:51.500479 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:51.500448 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lth6t"] Nov 02 22:54:51.501277 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:51.500679 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-lth6t" podUID="dcea57b2-2614-41a6-a4a9-7d40c86732a6" containerName="authorino" containerID="cri-o://791421f61642836a8169d4d450bdf16b029f1b0461254c8031674a67d0cbda01" gracePeriod=30 Nov 02 22:54:51.776140 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:51.776090 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-lth6t" Nov 02 22:54:51.878506 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:51.878475 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcea57b2-2614-41a6-a4a9-7d40c86732a6-kube-api-access-86chv\" (UniqueName: \"kubernetes.io/projected/dcea57b2-2614-41a6-a4a9-7d40c86732a6-kube-api-access-86chv\") pod \"dcea57b2-2614-41a6-a4a9-7d40c86732a6\" (UID: \"dcea57b2-2614-41a6-a4a9-7d40c86732a6\") " Nov 02 22:54:51.881299 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:51.881270 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcea57b2-2614-41a6-a4a9-7d40c86732a6-kube-api-access-86chv" pod "dcea57b2-2614-41a6-a4a9-7d40c86732a6" (UID: "dcea57b2-2614-41a6-a4a9-7d40c86732a6"). InnerVolumeSpecName "kube-api-access-86chv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:54:51.979564 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:51.979521 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86chv\" (UniqueName: \"kubernetes.io/projected/dcea57b2-2614-41a6-a4a9-7d40c86732a6-kube-api-access-86chv\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:54:52.065397 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:52.065367 5727 generic.go:356] "Generic (PLEG): container finished" podID="dcea57b2-2614-41a6-a4a9-7d40c86732a6" containerID="791421f61642836a8169d4d450bdf16b029f1b0461254c8031674a67d0cbda01" exitCode=0 Nov 02 22:54:52.065572 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:52.065416 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-lth6t" Nov 02 22:54:52.065572 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:52.065427 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-lth6t" event={"ID":"dcea57b2-2614-41a6-a4a9-7d40c86732a6","Type":"ContainerDied","Data":"791421f61642836a8169d4d450bdf16b029f1b0461254c8031674a67d0cbda01"} Nov 02 22:54:52.065572 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:52.065498 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-lth6t" event={"ID":"dcea57b2-2614-41a6-a4a9-7d40c86732a6","Type":"ContainerDied","Data":"b26508cd53304c49900d60add9c1842325ffd8ead2fc997c1a5db92a81ba1285"} Nov 02 22:54:52.065572 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:52.065528 5727 scope.go:122] "RemoveContainer" containerID="791421f61642836a8169d4d450bdf16b029f1b0461254c8031674a67d0cbda01" Nov 02 22:54:52.076003 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:52.075970 5727 scope.go:122] "RemoveContainer" containerID="791421f61642836a8169d4d450bdf16b029f1b0461254c8031674a67d0cbda01" Nov 02 22:54:52.076311 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:54:52.076273 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"791421f61642836a8169d4d450bdf16b029f1b0461254c8031674a67d0cbda01\": container with ID starting with 791421f61642836a8169d4d450bdf16b029f1b0461254c8031674a67d0cbda01 not found: ID does not exist" containerID="791421f61642836a8169d4d450bdf16b029f1b0461254c8031674a67d0cbda01" Nov 02 22:54:52.076420 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:52.076314 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"791421f61642836a8169d4d450bdf16b029f1b0461254c8031674a67d0cbda01"} err="failed to get container status \"791421f61642836a8169d4d450bdf16b029f1b0461254c8031674a67d0cbda01\": rpc error: code = NotFound desc = could not find container \"791421f61642836a8169d4d450bdf16b029f1b0461254c8031674a67d0cbda01\": container with ID starting with 791421f61642836a8169d4d450bdf16b029f1b0461254c8031674a67d0cbda01 not found: ID does not exist" Nov 02 22:54:52.228944 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:52.228906 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lth6t"] Nov 02 22:54:52.231000 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:52.230978 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lth6t"] Nov 02 22:54:52.458822 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:52.458740 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-knpxk"] Nov 02 22:54:52.459251 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:52.459234 5727 cpu_manager.go:414] "containerMap: removing container" podUID="dcea57b2-2614-41a6-a4a9-7d40c86732a6" containerName="authorino" Nov 02 22:54:52.459251 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:52.459252 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcea57b2-2614-41a6-a4a9-7d40c86732a6" containerName="authorino" Nov 02 22:54:52.459399 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:52.459328 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcea57b2-2614-41a6-a4a9-7d40c86732a6" containerName="authorino" Nov 02 22:54:52.472952 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:52.472932 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-knpxk" Nov 02 22:54:52.484000 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:52.483978 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f601f493-313e-40b5-9e89-0c492a35255a-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-knpxk\" (UID: \"f601f493-313e-40b5-9e89-0c492a35255a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-knpxk" Nov 02 22:54:52.484114 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:52.484021 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f601f493-313e-40b5-9e89-0c492a35255a-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-knpxk\" (UID: \"f601f493-313e-40b5-9e89-0c492a35255a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-knpxk" Nov 02 22:54:52.484114 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:52.484093 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f601f493-313e-40b5-9e89-0c492a35255a-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-knpxk\" (UID: \"f601f493-313e-40b5-9e89-0c492a35255a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-knpxk" Nov 02 22:54:52.484260 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:52.484142 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7kx7\" (UniqueName: \"kubernetes.io/projected/f601f493-313e-40b5-9e89-0c492a35255a-kube-api-access-c7kx7\") pod \"kserve-module-controller-manager-7566cf5fc8-knpxk\" (UID: \"f601f493-313e-40b5-9e89-0c492a35255a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-knpxk" Nov 02 22:54:52.523642 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:52.523609 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-knpxk"] Nov 02 22:54:52.525890 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:52.525847 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-2wpqd\"" Nov 02 22:54:52.526025 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:52.525943 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 22:54:52.526099 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:52.526036 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 22:54:52.584629 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:52.584596 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f601f493-313e-40b5-9e89-0c492a35255a-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-knpxk\" (UID: \"f601f493-313e-40b5-9e89-0c492a35255a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-knpxk" Nov 02 22:54:52.584793 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:52.584634 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7kx7\" (UniqueName: \"kubernetes.io/projected/f601f493-313e-40b5-9e89-0c492a35255a-kube-api-access-c7kx7\") pod \"kserve-module-controller-manager-7566cf5fc8-knpxk\" (UID: \"f601f493-313e-40b5-9e89-0c492a35255a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-knpxk" Nov 02 22:54:52.584793 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:52.584686 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f601f493-313e-40b5-9e89-0c492a35255a-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-knpxk\" (UID: \"f601f493-313e-40b5-9e89-0c492a35255a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-knpxk" Nov 02 22:54:52.584793 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:52.584711 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f601f493-313e-40b5-9e89-0c492a35255a-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-knpxk\" (UID: \"f601f493-313e-40b5-9e89-0c492a35255a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-knpxk" Nov 02 22:54:52.585035 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:52.585013 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f601f493-313e-40b5-9e89-0c492a35255a-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-knpxk\" (UID: \"f601f493-313e-40b5-9e89-0c492a35255a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-knpxk" Nov 02 22:54:52.585035 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:52.585026 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f601f493-313e-40b5-9e89-0c492a35255a-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-knpxk\" (UID: \"f601f493-313e-40b5-9e89-0c492a35255a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-knpxk" Nov 02 22:54:52.620285 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:52.620253 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 02 22:54:52.626252 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:52.626225 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f601f493-313e-40b5-9e89-0c492a35255a-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-knpxk\" (UID: \"f601f493-313e-40b5-9e89-0c492a35255a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-knpxk" Nov 02 22:54:52.648263 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:52.648238 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7kx7\" (UniqueName: \"kubernetes.io/projected/f601f493-313e-40b5-9e89-0c492a35255a-kube-api-access-c7kx7\") pod \"kserve-module-controller-manager-7566cf5fc8-knpxk\" (UID: \"f601f493-313e-40b5-9e89-0c492a35255a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-knpxk" Nov 02 22:54:52.791117 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:52.791036 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-knpxk" Nov 02 22:54:53.236812 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:53.236787 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-knpxk"] Nov 02 22:54:53.238874 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:54:53.238838 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf601f493_313e_40b5_9e89_0c492a35255a.slice/crio-fd52e6a5d3cceba32682e6eb8b0efd4cb72c144c53e307ff9b274a144e189fbf WatchSource:0}: Error finding container fd52e6a5d3cceba32682e6eb8b0efd4cb72c144c53e307ff9b274a144e189fbf: Status 404 returned error can't find the container with id fd52e6a5d3cceba32682e6eb8b0efd4cb72c144c53e307ff9b274a144e189fbf Nov 02 22:54:53.882619 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:53.882581 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcea57b2-2614-41a6-a4a9-7d40c86732a6" path="/var/lib/kubelet/pods/dcea57b2-2614-41a6-a4a9-7d40c86732a6/volumes" Nov 02 22:54:54.079892 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:54.079862 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-knpxk" event={"ID":"f601f493-313e-40b5-9e89-0c492a35255a","Type":"ContainerStarted","Data":"fd52e6a5d3cceba32682e6eb8b0efd4cb72c144c53e307ff9b274a144e189fbf"} Nov 02 22:54:54.081444 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:54.081416 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr9qg" event={"ID":"1394c83b-9059-4e3e-928f-bde172669b09","Type":"ContainerStarted","Data":"2c93de184f320da99cf6cd95fe34883c0cc94a740cc2243ad1e8bde9353e640a"} Nov 02 22:54:54.197496 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:54.197442 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr9qg" podStartSLOduration=2.3480246080000002 podStartE2EDuration="8.197425096s" totalImagesPullingTime="5.849400488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:54:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:54:54.195295438 +0000 UTC m=+380.433793119" watchObservedRunningTime="2026-11-02 22:54:54.197425096 +0000 UTC m=+380.435922758" Nov 02 22:54:57.100396 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:57.100359 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-knpxk" event={"ID":"f601f493-313e-40b5-9e89-0c492a35255a","Type":"ContainerStarted","Data":"fc07e80d08a6a971f7b60fc51d2839ca4ed3ae3bf63cb971d97910cc5f3aa035"} Nov 02 22:54:57.100806 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:57.100444 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-knpxk" Nov 02 22:54:57.317260 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:54:57.317199 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-knpxk" podStartSLOduration=2.510494802 podStartE2EDuration="5.317185775s" totalImagesPullingTime="2.806690973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:54:57.267520804 +0000 UTC m=+383.506018466" watchObservedRunningTime="2026-11-02 22:54:57.317185775 +0000 UTC m=+383.555683437" Nov 02 22:54:57.844165 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:54:57.844108 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 22:55:18.106373 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:55:18.106342 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-knpxk" Nov 02 22:55:27.844670 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:55:27.844635 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 22:55:35.164342 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:55:35.164307 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cb6tl"] Nov 02 22:55:35.184748 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:55:35.184716 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cb6tl" Nov 02 22:55:35.225912 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:55:35.225869 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cb6tl"] Nov 02 22:55:35.273174 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:55:35.273096 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 02 22:55:35.273174 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:55:35.273108 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 02 22:55:35.273431 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:55:35.273379 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 02 22:55:35.273715 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:55:35.273695 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-zhffl\"" Nov 02 22:55:35.372917 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:55:35.372887 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/54f62e96-134f-44b3-839e-a02edd85ab08-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cb6tl\" (UID: \"54f62e96-134f-44b3-839e-a02edd85ab08\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cb6tl" Nov 02 22:55:35.373120 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:55:35.372921 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54f62e96-134f-44b3-839e-a02edd85ab08-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cb6tl\" (UID: \"54f62e96-134f-44b3-839e-a02edd85ab08\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cb6tl" Nov 02 22:55:35.373120 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:55:35.373096 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b949z\" (UniqueName: \"kubernetes.io/projected/54f62e96-134f-44b3-839e-a02edd85ab08-kube-api-access-b949z\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cb6tl\" (UID: \"54f62e96-134f-44b3-839e-a02edd85ab08\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cb6tl" Nov 02 22:55:35.373262 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:55:35.373162 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/54f62e96-134f-44b3-839e-a02edd85ab08-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cb6tl\" (UID: \"54f62e96-134f-44b3-839e-a02edd85ab08\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cb6tl" Nov 02 22:55:35.473679 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:55:35.473588 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/54f62e96-134f-44b3-839e-a02edd85ab08-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cb6tl\" (UID: \"54f62e96-134f-44b3-839e-a02edd85ab08\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cb6tl" Nov 02 22:55:35.473679 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:55:35.473628 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54f62e96-134f-44b3-839e-a02edd85ab08-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cb6tl\" (UID: \"54f62e96-134f-44b3-839e-a02edd85ab08\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cb6tl" Nov 02 22:55:35.473903 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:55:35.473702 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b949z\" (UniqueName: \"kubernetes.io/projected/54f62e96-134f-44b3-839e-a02edd85ab08-kube-api-access-b949z\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cb6tl\" (UID: \"54f62e96-134f-44b3-839e-a02edd85ab08\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cb6tl" Nov 02 22:55:35.473903 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:55:35.473730 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/54f62e96-134f-44b3-839e-a02edd85ab08-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cb6tl\" (UID: \"54f62e96-134f-44b3-839e-a02edd85ab08\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cb6tl" Nov 02 22:55:35.473903 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:55:35.473787 5727 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Nov 02 22:55:35.473903 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:55:35.473829 5727 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 02 22:55:35.473903 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:55:35.473876 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54f62e96-134f-44b3-839e-a02edd85ab08-tls-cert podName:54f62e96-134f-44b3-839e-a02edd85ab08 nodeName:}" failed. No retries permitted until 2026-11-02 22:55:35.973852151 +0000 UTC m=+422.212349807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/54f62e96-134f-44b3-839e-a02edd85ab08-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-cb6tl" (UID: "54f62e96-134f-44b3-839e-a02edd85ab08") : secret "usage-logs-tenancy-proxy-tls" not found Nov 02 22:55:35.473903 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:55:35.473902 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/54f62e96-134f-44b3-839e-a02edd85ab08-proxy-source podName:54f62e96-134f-44b3-839e-a02edd85ab08 nodeName:}" failed. No retries permitted until 2026-11-02 22:55:35.973885009 +0000 UTC m=+422.212382650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/54f62e96-134f-44b3-839e-a02edd85ab08-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-cb6tl" (UID: "54f62e96-134f-44b3-839e-a02edd85ab08") : configmap "usage-tenancy-proxy-source" not found Nov 02 22:55:35.474354 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:55:35.474336 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54f62e96-134f-44b3-839e-a02edd85ab08-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cb6tl\" (UID: \"54f62e96-134f-44b3-839e-a02edd85ab08\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cb6tl" Nov 02 22:55:35.539979 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:55:35.539947 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b949z\" (UniqueName: \"kubernetes.io/projected/54f62e96-134f-44b3-839e-a02edd85ab08-kube-api-access-b949z\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cb6tl\" (UID: \"54f62e96-134f-44b3-839e-a02edd85ab08\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cb6tl" Nov 02 22:55:35.977788 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:55:35.977752 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/54f62e96-134f-44b3-839e-a02edd85ab08-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cb6tl\" (UID: \"54f62e96-134f-44b3-839e-a02edd85ab08\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cb6tl" Nov 02 22:55:35.978001 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:55:35.977800 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/54f62e96-134f-44b3-839e-a02edd85ab08-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cb6tl\" (UID: \"54f62e96-134f-44b3-839e-a02edd85ab08\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cb6tl" Nov 02 22:55:35.978653 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:55:35.978628 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/54f62e96-134f-44b3-839e-a02edd85ab08-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cb6tl\" (UID: \"54f62e96-134f-44b3-839e-a02edd85ab08\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cb6tl" Nov 02 22:55:35.980349 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:55:35.980330 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/54f62e96-134f-44b3-839e-a02edd85ab08-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cb6tl\" (UID: \"54f62e96-134f-44b3-839e-a02edd85ab08\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cb6tl" Nov 02 22:55:36.098021 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:55:36.097981 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cb6tl" Nov 02 22:55:36.477821 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:55:36.477800 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cb6tl"] Nov 02 22:55:36.480388 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:55:36.480355 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f62e96_134f_44b3_839e_a02edd85ab08.slice/crio-dcd63e2700506a22c02738a8476ecf45633f9d3c337648708542d7bb08c963a1 WatchSource:0}: Error finding container dcd63e2700506a22c02738a8476ecf45633f9d3c337648708542d7bb08c963a1: Status 404 returned error can't find the container with id dcd63e2700506a22c02738a8476ecf45633f9d3c337648708542d7bb08c963a1 Nov 02 22:55:37.308958 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:55:37.308923 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cb6tl" event={"ID":"54f62e96-134f-44b3-839e-a02edd85ab08","Type":"ContainerStarted","Data":"dcd63e2700506a22c02738a8476ecf45633f9d3c337648708542d7bb08c963a1"} Nov 02 22:55:49.391372 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:55:49.391333 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cb6tl" event={"ID":"54f62e96-134f-44b3-839e-a02edd85ab08","Type":"ContainerStarted","Data":"56cc14900ef357009f8aaff723c21fb2cab15fc01307dc7f517f29e1523ca7e4"} Nov 02 22:55:49.391809 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:55:49.391449 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cb6tl" Nov 02 22:55:49.510655 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:55:49.510567 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cb6tl" podStartSLOduration=1.765038734 podStartE2EDuration="14.510554613s" totalImagesPullingTime="12.745515879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:55:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:55:49.51019497 +0000 UTC m=+435.748692652" watchObservedRunningTime="2026-11-02 22:55:49.510554613 +0000 UTC m=+435.749052275" Nov 02 22:55:57.844966 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:55:57.844924 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 22:56:00.334027 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.333994 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-qprz9"] Nov 02 22:56:00.452586 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.452552 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cb6tl" Nov 02 22:56:00.452757 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.452594 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-qprz9"] Nov 02 22:56:00.452757 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.452624 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-vmxhx"] Nov 02 22:56:00.452871 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.452855 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-qprz9" Nov 02 22:56:00.483945 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.483916 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmxhx" Nov 02 22:56:00.486200 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.486177 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-vmxhx"] Nov 02 22:56:00.499473 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.499446 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzqd6\" (UniqueName: \"kubernetes.io/projected/b7266a2f-427b-47f5-9528-e8c111ae8d04-kube-api-access-pzqd6\") pod \"kserve-controller-manager-776cc66d47-qprz9\" (UID: \"b7266a2f-427b-47f5-9528-e8c111ae8d04\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qprz9" Nov 02 22:56:00.499737 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.499716 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b7266a2f-427b-47f5-9528-e8c111ae8d04-cert\") pod \"kserve-controller-manager-776cc66d47-qprz9\" (UID: \"b7266a2f-427b-47f5-9528-e8c111ae8d04\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qprz9" Nov 02 22:56:00.533240 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.533218 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-9wvlx\"" Nov 02 22:56:00.533381 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.533363 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 02 22:56:00.555778 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.555757 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-cnzpg"] Nov 02 22:56:00.577343 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.577315 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-cnzpg" Nov 02 22:56:00.600280 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.600255 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzqd6\" (UniqueName: \"kubernetes.io/projected/b7266a2f-427b-47f5-9528-e8c111ae8d04-kube-api-access-pzqd6\") pod \"kserve-controller-manager-776cc66d47-qprz9\" (UID: \"b7266a2f-427b-47f5-9528-e8c111ae8d04\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qprz9" Nov 02 22:56:00.600411 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.600316 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/724a86df-d3a5-412d-afb9-becaa9cfd2cd-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-vmxhx\" (UID: \"724a86df-d3a5-412d-afb9-becaa9cfd2cd\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmxhx" Nov 02 22:56:00.600411 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.600353 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqdbx\" (UniqueName: \"kubernetes.io/projected/724a86df-d3a5-412d-afb9-becaa9cfd2cd-kube-api-access-bqdbx\") pod \"llmisvc-controller-manager-869d5dcc5d-vmxhx\" (UID: \"724a86df-d3a5-412d-afb9-becaa9cfd2cd\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmxhx" Nov 02 22:56:00.600411 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.600406 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b7266a2f-427b-47f5-9528-e8c111ae8d04-cert\") pod \"kserve-controller-manager-776cc66d47-qprz9\" (UID: \"b7266a2f-427b-47f5-9528-e8c111ae8d04\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qprz9" Nov 02 22:56:00.603523 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.603497 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b7266a2f-427b-47f5-9528-e8c111ae8d04-cert\") pod \"kserve-controller-manager-776cc66d47-qprz9\" (UID: \"b7266a2f-427b-47f5-9528-e8c111ae8d04\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qprz9" Nov 02 22:56:00.603914 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.603894 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-cnzpg"] Nov 02 22:56:00.605207 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.605189 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-mjzwb\"" Nov 02 22:56:00.605363 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.605267 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 02 22:56:00.661231 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.661201 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzqd6\" (UniqueName: \"kubernetes.io/projected/b7266a2f-427b-47f5-9528-e8c111ae8d04-kube-api-access-pzqd6\") pod \"kserve-controller-manager-776cc66d47-qprz9\" (UID: \"b7266a2f-427b-47f5-9528-e8c111ae8d04\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qprz9" Nov 02 22:56:00.701271 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.701243 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b86548a3-cf54-41a9-8c65-f543ae9daa83-tls-certs\") pod \"model-serving-api-75bd89fdfc-cnzpg\" (UID: \"b86548a3-cf54-41a9-8c65-f543ae9daa83\") " pod="opendatahub/model-serving-api-75bd89fdfc-cnzpg" Nov 02 22:56:00.701421 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.701279 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rhpd\" (UniqueName: \"kubernetes.io/projected/b86548a3-cf54-41a9-8c65-f543ae9daa83-kube-api-access-2rhpd\") pod \"model-serving-api-75bd89fdfc-cnzpg\" (UID: \"b86548a3-cf54-41a9-8c65-f543ae9daa83\") " pod="opendatahub/model-serving-api-75bd89fdfc-cnzpg" Nov 02 22:56:00.701421 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.701311 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/724a86df-d3a5-412d-afb9-becaa9cfd2cd-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-vmxhx\" (UID: \"724a86df-d3a5-412d-afb9-becaa9cfd2cd\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmxhx" Nov 02 22:56:00.701501 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.701479 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqdbx\" (UniqueName: \"kubernetes.io/projected/724a86df-d3a5-412d-afb9-becaa9cfd2cd-kube-api-access-bqdbx\") pod \"llmisvc-controller-manager-869d5dcc5d-vmxhx\" (UID: \"724a86df-d3a5-412d-afb9-becaa9cfd2cd\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmxhx" Nov 02 22:56:00.703778 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.703759 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/724a86df-d3a5-412d-afb9-becaa9cfd2cd-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-vmxhx\" (UID: \"724a86df-d3a5-412d-afb9-becaa9cfd2cd\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmxhx" Nov 02 22:56:00.705176 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.705160 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 02 22:56:00.705247 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.705177 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-jnxpk\"" Nov 02 22:56:00.761784 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.761762 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqdbx\" (UniqueName: \"kubernetes.io/projected/724a86df-d3a5-412d-afb9-becaa9cfd2cd-kube-api-access-bqdbx\") pod \"llmisvc-controller-manager-869d5dcc5d-vmxhx\" (UID: \"724a86df-d3a5-412d-afb9-becaa9cfd2cd\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmxhx" Nov 02 22:56:00.792228 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.792208 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-qprz9" Nov 02 22:56:00.801892 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.801871 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b86548a3-cf54-41a9-8c65-f543ae9daa83-tls-certs\") pod \"model-serving-api-75bd89fdfc-cnzpg\" (UID: \"b86548a3-cf54-41a9-8c65-f543ae9daa83\") " pod="opendatahub/model-serving-api-75bd89fdfc-cnzpg" Nov 02 22:56:00.801985 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.801902 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rhpd\" (UniqueName: \"kubernetes.io/projected/b86548a3-cf54-41a9-8c65-f543ae9daa83-kube-api-access-2rhpd\") pod \"model-serving-api-75bd89fdfc-cnzpg\" (UID: \"b86548a3-cf54-41a9-8c65-f543ae9daa83\") " pod="opendatahub/model-serving-api-75bd89fdfc-cnzpg" Nov 02 22:56:00.804231 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.804215 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b86548a3-cf54-41a9-8c65-f543ae9daa83-tls-certs\") pod \"model-serving-api-75bd89fdfc-cnzpg\" (UID: \"b86548a3-cf54-41a9-8c65-f543ae9daa83\") " pod="opendatahub/model-serving-api-75bd89fdfc-cnzpg" Nov 02 22:56:00.821248 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.821230 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmxhx" Nov 02 22:56:00.862632 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.862585 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rhpd\" (UniqueName: \"kubernetes.io/projected/b86548a3-cf54-41a9-8c65-f543ae9daa83-kube-api-access-2rhpd\") pod \"model-serving-api-75bd89fdfc-cnzpg\" (UID: \"b86548a3-cf54-41a9-8c65-f543ae9daa83\") " pod="opendatahub/model-serving-api-75bd89fdfc-cnzpg" Nov 02 22:56:00.906800 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:00.906600 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-cnzpg" Nov 02 22:56:01.197978 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:01.197956 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-qprz9"] Nov 02 22:56:01.199994 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:56:01.199963 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7266a2f_427b_47f5_9528_e8c111ae8d04.slice/crio-e9e22aed891f3bc8838ecc9a4f484686781f91f52979b10731370df5e647187f WatchSource:0}: Error finding container e9e22aed891f3bc8838ecc9a4f484686781f91f52979b10731370df5e647187f: Status 404 returned error can't find the container with id e9e22aed891f3bc8838ecc9a4f484686781f91f52979b10731370df5e647187f Nov 02 22:56:01.248454 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:01.248432 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-vmxhx"] Nov 02 22:56:01.249690 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:56:01.249650 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod724a86df_d3a5_412d_afb9_becaa9cfd2cd.slice/crio-c5b3861dbc288837fa523601cfe2e8952924fc5a4b3d583e8b9373f48a19bb64 WatchSource:0}: Error finding container c5b3861dbc288837fa523601cfe2e8952924fc5a4b3d583e8b9373f48a19bb64: Status 404 returned error can't find the container with id c5b3861dbc288837fa523601cfe2e8952924fc5a4b3d583e8b9373f48a19bb64 Nov 02 22:56:01.306744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:01.306722 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-cnzpg"] Nov 02 22:56:01.308072 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:56:01.308044 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb86548a3_cf54_41a9_8c65_f543ae9daa83.slice/crio-4f0d1b1fa6356263f5e4112c287dbe2f7cdeb1a283ce679c74b5b5a7ecdb11a9 WatchSource:0}: Error finding container 4f0d1b1fa6356263f5e4112c287dbe2f7cdeb1a283ce679c74b5b5a7ecdb11a9: Status 404 returned error can't find the container with id 4f0d1b1fa6356263f5e4112c287dbe2f7cdeb1a283ce679c74b5b5a7ecdb11a9 Nov 02 22:56:01.457038 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:01.456959 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmxhx" event={"ID":"724a86df-d3a5-412d-afb9-becaa9cfd2cd","Type":"ContainerStarted","Data":"c5b3861dbc288837fa523601cfe2e8952924fc5a4b3d583e8b9373f48a19bb64"} Nov 02 22:56:01.458213 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:01.458190 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-qprz9" event={"ID":"b7266a2f-427b-47f5-9528-e8c111ae8d04","Type":"ContainerStarted","Data":"e9e22aed891f3bc8838ecc9a4f484686781f91f52979b10731370df5e647187f"} Nov 02 22:56:01.459437 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:01.459416 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-cnzpg" event={"ID":"b86548a3-cf54-41a9-8c65-f543ae9daa83","Type":"ContainerStarted","Data":"4f0d1b1fa6356263f5e4112c287dbe2f7cdeb1a283ce679c74b5b5a7ecdb11a9"} Nov 02 22:56:05.494640 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:05.494603 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-cnzpg" event={"ID":"b86548a3-cf54-41a9-8c65-f543ae9daa83","Type":"ContainerStarted","Data":"5ac3b9132620ee1b870986d235d89bd8852d6ebd9feae5c05b9ef7f0159416b2"} Nov 02 22:56:05.495104 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:05.494675 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-cnzpg" Nov 02 22:56:05.612523 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:05.612458 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-cnzpg" podStartSLOduration=2.5096278229999998 podStartE2EDuration="5.612440804s" totalImagesPullingTime="3.102812981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:56:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:56:05.611974976 +0000 UTC m=+451.850472672" watchObservedRunningTime="2026-11-02 22:56:05.612440804 +0000 UTC m=+451.850938466" Nov 02 22:56:06.503011 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:06.502974 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmxhx" event={"ID":"724a86df-d3a5-412d-afb9-becaa9cfd2cd","Type":"ContainerStarted","Data":"a76f50a037e4405d3940002a5034562f66511a1cbf8339327a16f6ab3aad31a6"} Nov 02 22:56:06.503527 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:06.503053 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmxhx" Nov 02 22:56:06.621005 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:06.620950 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmxhx" podStartSLOduration=1.947766073 podStartE2EDuration="6.620930484s" totalImagesPullingTime="4.673164411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:56:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:56:06.617921492 +0000 UTC m=+452.856419174" watchObservedRunningTime="2026-11-02 22:56:06.620930484 +0000 UTC m=+452.859428149" Nov 02 22:56:07.509559 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:07.509524 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-qprz9" event={"ID":"b7266a2f-427b-47f5-9528-e8c111ae8d04","Type":"ContainerStarted","Data":"c455709d25c5271fad440ed4afe4df93635a6d64e704c2bd5d0f34e2960a5d45"} Nov 02 22:56:07.510040 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:07.509704 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-qprz9" Nov 02 22:56:07.625626 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:07.625574 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-qprz9" podStartSLOduration=1.791227899 podStartE2EDuration="7.62556009s" totalImagesPullingTime="5.834332191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:56:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:56:07.623895875 +0000 UTC m=+453.862393537" watchObservedRunningTime="2026-11-02 22:56:07.62556009 +0000 UTC m=+453.864057752" Nov 02 22:56:16.507688 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:16.507600 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-cnzpg" Nov 02 22:56:20.387889 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:20.387860 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-4bn4s"] Nov 02 22:56:20.390322 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:20.390307 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-4bn4s" Nov 02 22:56:20.446032 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:20.446002 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-4bn4s"] Nov 02 22:56:20.478305 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:20.478280 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v55wr\" (UniqueName: \"kubernetes.io/projected/b6d5dfd6-5a95-4dee-97fe-209ffe1ed0e8-kube-api-access-v55wr\") pod \"authorino-6f6944bcdd-4bn4s\" (UID: \"b6d5dfd6-5a95-4dee-97fe-209ffe1ed0e8\") " pod="kuadrant-system/authorino-6f6944bcdd-4bn4s" Nov 02 22:56:20.492182 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:20.492163 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dpz6v\"" Nov 02 22:56:20.579153 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:20.579105 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v55wr\" (UniqueName: \"kubernetes.io/projected/b6d5dfd6-5a95-4dee-97fe-209ffe1ed0e8-kube-api-access-v55wr\") pod \"authorino-6f6944bcdd-4bn4s\" (UID: \"b6d5dfd6-5a95-4dee-97fe-209ffe1ed0e8\") " pod="kuadrant-system/authorino-6f6944bcdd-4bn4s" Nov 02 22:56:20.602403 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:20.602376 5727 generic.go:356] "Generic (PLEG): container finished" podID="c4c2f4f5-743d-4e9d-ae63-1769c521b4ad" containerID="b0541cd249a21e0f7d46bdf6006d64ab830a350f3213ea0df81c1a2234d21d47" exitCode=0 Nov 02 22:56:20.602551 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:20.602451 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" event={"ID":"c4c2f4f5-743d-4e9d-ae63-1769c521b4ad","Type":"ContainerDied","Data":"b0541cd249a21e0f7d46bdf6006d64ab830a350f3213ea0df81c1a2234d21d47"} Nov 02 22:56:20.602805 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:20.602787 5727 scope.go:122] "RemoveContainer" containerID="b0541cd249a21e0f7d46bdf6006d64ab830a350f3213ea0df81c1a2234d21d47" Nov 02 22:56:20.604013 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:20.603980 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 22:56:20.604103 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:20.604069 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 22:56:20.604204 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:20.604118 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 22:56:20.640121 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:20.640074 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v55wr\" (UniqueName: \"kubernetes.io/projected/b6d5dfd6-5a95-4dee-97fe-209ffe1ed0e8-kube-api-access-v55wr\") pod \"authorino-6f6944bcdd-4bn4s\" (UID: \"b6d5dfd6-5a95-4dee-97fe-209ffe1ed0e8\") " pod="kuadrant-system/authorino-6f6944bcdd-4bn4s" Nov 02 22:56:20.701837 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:20.701810 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-4bn4s" Nov 02 22:56:20.768178 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:20.768145 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-4bn4s"] Nov 02 22:56:20.788506 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:20.788467 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5959c88d68-gdn7l"] Nov 02 22:56:20.793520 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:20.793492 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5959c88d68-gdn7l" Nov 02 22:56:20.845790 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:20.845761 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5959c88d68-gdn7l"] Nov 02 22:56:20.882524 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:20.882491 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtwjd\" (UniqueName: \"kubernetes.io/projected/add45684-e452-47db-ae10-dc3a74827fd3-kube-api-access-wtwjd\") pod \"authorino-5959c88d68-gdn7l\" (UID: \"add45684-e452-47db-ae10-dc3a74827fd3\") " pod="kuadrant-system/authorino-5959c88d68-gdn7l" Nov 02 22:56:20.983672 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:20.983588 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtwjd\" (UniqueName: \"kubernetes.io/projected/add45684-e452-47db-ae10-dc3a74827fd3-kube-api-access-wtwjd\") pod \"authorino-5959c88d68-gdn7l\" (UID: \"add45684-e452-47db-ae10-dc3a74827fd3\") " pod="kuadrant-system/authorino-5959c88d68-gdn7l" Nov 02 22:56:21.044824 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:21.044787 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtwjd\" (UniqueName: \"kubernetes.io/projected/add45684-e452-47db-ae10-dc3a74827fd3-kube-api-access-wtwjd\") pod \"authorino-5959c88d68-gdn7l\" (UID: \"add45684-e452-47db-ae10-dc3a74827fd3\") " pod="kuadrant-system/authorino-5959c88d68-gdn7l" Nov 02 22:56:21.084330 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:21.084303 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-4bn4s"] Nov 02 22:56:21.086216 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:56:21.086187 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6d5dfd6_5a95_4dee_97fe_209ffe1ed0e8.slice/crio-8e8cdc9e6f9c491adeb20b40c727b949a58ad482ec202065920bc4a5d4482dd5 WatchSource:0}: Error finding container 8e8cdc9e6f9c491adeb20b40c727b949a58ad482ec202065920bc4a5d4482dd5: Status 404 returned error can't find the container with id 8e8cdc9e6f9c491adeb20b40c727b949a58ad482ec202065920bc4a5d4482dd5 Nov 02 22:56:21.112065 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:21.112042 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5959c88d68-gdn7l" Nov 02 22:56:21.227103 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:21.227075 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5959c88d68-gdn7l"] Nov 02 22:56:21.277434 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:21.277349 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-78f5fdb9b6-bmxbs"] Nov 02 22:56:21.280471 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:21.280453 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78f5fdb9b6-bmxbs" Nov 02 22:56:21.317561 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:21.317538 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78f5fdb9b6-bmxbs"] Nov 02 22:56:21.335096 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:21.335066 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 02 22:56:21.387976 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:21.387941 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9cb33a7f-dd45-41ad-9b78-45bf297a29d8-tls-cert\") pod \"authorino-78f5fdb9b6-bmxbs\" (UID: \"9cb33a7f-dd45-41ad-9b78-45bf297a29d8\") " pod="kuadrant-system/authorino-78f5fdb9b6-bmxbs" Nov 02 22:56:21.387976 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:21.387986 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq7nb\" (UniqueName: \"kubernetes.io/projected/9cb33a7f-dd45-41ad-9b78-45bf297a29d8-kube-api-access-vq7nb\") pod \"authorino-78f5fdb9b6-bmxbs\" (UID: \"9cb33a7f-dd45-41ad-9b78-45bf297a29d8\") " pod="kuadrant-system/authorino-78f5fdb9b6-bmxbs" Nov 02 22:56:21.489434 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:21.489396 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9cb33a7f-dd45-41ad-9b78-45bf297a29d8-tls-cert\") pod \"authorino-78f5fdb9b6-bmxbs\" (UID: \"9cb33a7f-dd45-41ad-9b78-45bf297a29d8\") " pod="kuadrant-system/authorino-78f5fdb9b6-bmxbs" Nov 02 22:56:21.489434 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:21.489433 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq7nb\" (UniqueName: \"kubernetes.io/projected/9cb33a7f-dd45-41ad-9b78-45bf297a29d8-kube-api-access-vq7nb\") pod \"authorino-78f5fdb9b6-bmxbs\" (UID: \"9cb33a7f-dd45-41ad-9b78-45bf297a29d8\") " pod="kuadrant-system/authorino-78f5fdb9b6-bmxbs" Nov 02 22:56:21.492071 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:21.492042 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9cb33a7f-dd45-41ad-9b78-45bf297a29d8-tls-cert\") pod \"authorino-78f5fdb9b6-bmxbs\" (UID: \"9cb33a7f-dd45-41ad-9b78-45bf297a29d8\") " pod="kuadrant-system/authorino-78f5fdb9b6-bmxbs" Nov 02 22:56:21.504690 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:21.504664 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5959c88d68-gdn7l"] Nov 02 22:56:21.506397 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:56:21.506375 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podadd45684_e452_47db_ae10_dc3a74827fd3.slice/crio-3dbb50f855c9c1ea2a4200afc36118b3d14301f2dce5c7ba8778044daf956d3b WatchSource:0}: Error finding container 3dbb50f855c9c1ea2a4200afc36118b3d14301f2dce5c7ba8778044daf956d3b: Status 404 returned error can't find the container with id 3dbb50f855c9c1ea2a4200afc36118b3d14301f2dce5c7ba8778044daf956d3b Nov 02 22:56:21.572212 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:21.550107 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq7nb\" (UniqueName: \"kubernetes.io/projected/9cb33a7f-dd45-41ad-9b78-45bf297a29d8-kube-api-access-vq7nb\") pod \"authorino-78f5fdb9b6-bmxbs\" (UID: \"9cb33a7f-dd45-41ad-9b78-45bf297a29d8\") " pod="kuadrant-system/authorino-78f5fdb9b6-bmxbs" Nov 02 22:56:21.592772 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:21.592738 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78f5fdb9b6-bmxbs" Nov 02 22:56:21.609871 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:21.609845 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5959c88d68-gdn7l" event={"ID":"add45684-e452-47db-ae10-dc3a74827fd3","Type":"ContainerStarted","Data":"3dbb50f855c9c1ea2a4200afc36118b3d14301f2dce5c7ba8778044daf956d3b"} Nov 02 22:56:21.612180 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:21.612158 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" event={"ID":"c4c2f4f5-743d-4e9d-ae63-1769c521b4ad","Type":"ContainerStarted","Data":"a9509b8d334971c129213619784bcab0fc02801d377adf57925dfdb4d4693d0e"} Nov 02 22:56:21.613401 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:21.613375 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-4bn4s" event={"ID":"b6d5dfd6-5a95-4dee-97fe-209ffe1ed0e8","Type":"ContainerStarted","Data":"8e8cdc9e6f9c491adeb20b40c727b949a58ad482ec202065920bc4a5d4482dd5"} Nov 02 22:56:21.802562 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:21.802522 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:56:21.803778 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:21.803751 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-gcxrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Nov 02 22:56:21.803881 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:21.803801 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" podUID="c4c2f4f5-743d-4e9d-ae63-1769c521b4ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Nov 02 22:56:21.976176 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:21.975451 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78f5fdb9b6-bmxbs"] Nov 02 22:56:21.977828 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:56:21.977799 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9cb33a7f_dd45_41ad_9b78_45bf297a29d8.slice/crio-07cfc61808e97707611e8bfcdc1c9a1ca05719831d84c1d87e6e0b974020bef4 WatchSource:0}: Error finding container 07cfc61808e97707611e8bfcdc1c9a1ca05719831d84c1d87e6e0b974020bef4: Status 404 returned error can't find the container with id 07cfc61808e97707611e8bfcdc1c9a1ca05719831d84c1d87e6e0b974020bef4 Nov 02 22:56:22.237753 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:22.237677 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:56:22.620610 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:22.620578 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78f5fdb9b6-bmxbs" event={"ID":"9cb33a7f-dd45-41ad-9b78-45bf297a29d8","Type":"ContainerStarted","Data":"07cfc61808e97707611e8bfcdc1c9a1ca05719831d84c1d87e6e0b974020bef4"} Nov 02 22:56:22.803431 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:22.803402 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-gcxrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Nov 02 22:56:22.803551 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:22.803464 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" podUID="c4c2f4f5-743d-4e9d-ae63-1769c521b4ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Nov 02 22:56:23.631248 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:23.631194 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-4bn4s" event={"ID":"b6d5dfd6-5a95-4dee-97fe-209ffe1ed0e8","Type":"ContainerStarted","Data":"1658f55c0ad060b2eafb4957b547711ef2425e54b23c7088f56c4a3fc0815e69"} Nov 02 22:56:23.631663 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:23.631428 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-4bn4s" podUID="b6d5dfd6-5a95-4dee-97fe-209ffe1ed0e8" containerName="authorino" containerID="cri-o://1658f55c0ad060b2eafb4957b547711ef2425e54b23c7088f56c4a3fc0815e69" gracePeriod=30 Nov 02 22:56:23.643826 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:23.643790 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5959c88d68-gdn7l" event={"ID":"add45684-e452-47db-ae10-dc3a74827fd3","Type":"ContainerStarted","Data":"077daf174c0dd94358e6e1b6b2f7afed5d6dfc274426435f5713cefc0543e0bd"} Nov 02 22:56:23.644005 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:23.643933 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5959c88d68-gdn7l" podUID="add45684-e452-47db-ae10-dc3a74827fd3" containerName="authorino" containerID="cri-o://077daf174c0dd94358e6e1b6b2f7afed5d6dfc274426435f5713cefc0543e0bd" gracePeriod=30 Nov 02 22:56:23.647888 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:23.646436 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78f5fdb9b6-bmxbs" event={"ID":"9cb33a7f-dd45-41ad-9b78-45bf297a29d8","Type":"ContainerStarted","Data":"fcb9b338f67b5e3c00e7b4038ecabb8c8eb8f7e6a755e1712d89d98fb43e95d3"} Nov 02 22:56:23.746046 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:23.745988 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-4bn4s" podStartSLOduration=2.136539577 podStartE2EDuration="3.745973964s" totalImagesPullingTime="1.609434387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:56:23.745157312 +0000 UTC m=+469.983654997" watchObservedRunningTime="2026-11-02 22:56:23.745973964 +0000 UTC m=+469.984471625" Nov 02 22:56:23.803584 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:23.803549 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-gcxrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Nov 02 22:56:23.803719 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:23.803611 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" podUID="c4c2f4f5-743d-4e9d-ae63-1769c521b4ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Nov 02 22:56:23.995105 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:23.995077 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-4bn4s" Nov 02 22:56:23.998583 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:23.998561 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5959c88d68-gdn7l" Nov 02 22:56:24.002812 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:24.002766 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-78f5fdb9b6-bmxbs" podStartSLOduration=1.786360922 podStartE2EDuration="3.002752117s" totalImagesPullingTime="1.216391195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:56:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:56:23.886338163 +0000 UTC m=+470.124835833" watchObservedRunningTime="2026-11-02 22:56:24.002752117 +0000 UTC m=+470.241249779" Nov 02 22:56:24.003500 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:24.003457 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5959c88d68-gdn7l" podStartSLOduration=2.716170879 podStartE2EDuration="4.003448032s" totalImagesPullingTime="1.287277153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:56:24.001583901 +0000 UTC m=+470.240081563" watchObservedRunningTime="2026-11-02 22:56:24.003448032 +0000 UTC m=+470.241945691" Nov 02 22:56:24.115220 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:24.115191 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/add45684-e452-47db-ae10-dc3a74827fd3-kube-api-access-wtwjd\" (UniqueName: \"kubernetes.io/projected/add45684-e452-47db-ae10-dc3a74827fd3-kube-api-access-wtwjd\") pod \"add45684-e452-47db-ae10-dc3a74827fd3\" (UID: \"add45684-e452-47db-ae10-dc3a74827fd3\") " Nov 02 22:56:24.115387 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:24.115255 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6d5dfd6-5a95-4dee-97fe-209ffe1ed0e8-kube-api-access-v55wr\" (UniqueName: \"kubernetes.io/projected/b6d5dfd6-5a95-4dee-97fe-209ffe1ed0e8-kube-api-access-v55wr\") pod \"b6d5dfd6-5a95-4dee-97fe-209ffe1ed0e8\" (UID: \"b6d5dfd6-5a95-4dee-97fe-209ffe1ed0e8\") " Nov 02 22:56:24.117398 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:24.117365 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add45684-e452-47db-ae10-dc3a74827fd3-kube-api-access-wtwjd" pod "add45684-e452-47db-ae10-dc3a74827fd3" (UID: "add45684-e452-47db-ae10-dc3a74827fd3"). InnerVolumeSpecName "kube-api-access-wtwjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:56:24.117576 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:24.117537 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d5dfd6-5a95-4dee-97fe-209ffe1ed0e8-kube-api-access-v55wr" pod "b6d5dfd6-5a95-4dee-97fe-209ffe1ed0e8" (UID: "b6d5dfd6-5a95-4dee-97fe-209ffe1ed0e8"). InnerVolumeSpecName "kube-api-access-v55wr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:56:24.216672 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:24.216642 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtwjd\" (UniqueName: \"kubernetes.io/projected/add45684-e452-47db-ae10-dc3a74827fd3-kube-api-access-wtwjd\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:56:24.216672 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:24.216672 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v55wr\" (UniqueName: \"kubernetes.io/projected/b6d5dfd6-5a95-4dee-97fe-209ffe1ed0e8-kube-api-access-v55wr\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:56:24.656154 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:24.655962 5727 generic.go:356] "Generic (PLEG): container finished" podID="add45684-e452-47db-ae10-dc3a74827fd3" containerID="077daf174c0dd94358e6e1b6b2f7afed5d6dfc274426435f5713cefc0543e0bd" exitCode=0 Nov 02 22:56:24.656671 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:24.656181 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5959c88d68-gdn7l" event={"ID":"add45684-e452-47db-ae10-dc3a74827fd3","Type":"ContainerDied","Data":"077daf174c0dd94358e6e1b6b2f7afed5d6dfc274426435f5713cefc0543e0bd"} Nov 02 22:56:24.656671 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:24.656242 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5959c88d68-gdn7l" event={"ID":"add45684-e452-47db-ae10-dc3a74827fd3","Type":"ContainerDied","Data":"3dbb50f855c9c1ea2a4200afc36118b3d14301f2dce5c7ba8778044daf956d3b"} Nov 02 22:56:24.656671 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:24.656271 5727 scope.go:122] "RemoveContainer" containerID="077daf174c0dd94358e6e1b6b2f7afed5d6dfc274426435f5713cefc0543e0bd" Nov 02 22:56:24.656671 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:24.656505 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5959c88d68-gdn7l" Nov 02 22:56:24.679508 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:24.668748 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-4bn4s" event={"ID":"b6d5dfd6-5a95-4dee-97fe-209ffe1ed0e8","Type":"ContainerDied","Data":"1658f55c0ad060b2eafb4957b547711ef2425e54b23c7088f56c4a3fc0815e69"} Nov 02 22:56:24.679508 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:24.668828 5727 generic.go:356] "Generic (PLEG): container finished" podID="b6d5dfd6-5a95-4dee-97fe-209ffe1ed0e8" containerID="1658f55c0ad060b2eafb4957b547711ef2425e54b23c7088f56c4a3fc0815e69" exitCode=0 Nov 02 22:56:24.679508 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:24.668911 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-4bn4s" event={"ID":"b6d5dfd6-5a95-4dee-97fe-209ffe1ed0e8","Type":"ContainerDied","Data":"8e8cdc9e6f9c491adeb20b40c727b949a58ad482ec202065920bc4a5d4482dd5"} Nov 02 22:56:24.679508 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:24.668646 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-4bn4s" Nov 02 22:56:24.687054 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:24.687033 5727 scope.go:122] "RemoveContainer" containerID="077daf174c0dd94358e6e1b6b2f7afed5d6dfc274426435f5713cefc0543e0bd" Nov 02 22:56:24.687489 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:56:24.687462 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"077daf174c0dd94358e6e1b6b2f7afed5d6dfc274426435f5713cefc0543e0bd\": container with ID starting with 077daf174c0dd94358e6e1b6b2f7afed5d6dfc274426435f5713cefc0543e0bd not found: ID does not exist" containerID="077daf174c0dd94358e6e1b6b2f7afed5d6dfc274426435f5713cefc0543e0bd" Nov 02 22:56:24.687577 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:24.687501 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"077daf174c0dd94358e6e1b6b2f7afed5d6dfc274426435f5713cefc0543e0bd"} err="failed to get container status \"077daf174c0dd94358e6e1b6b2f7afed5d6dfc274426435f5713cefc0543e0bd\": rpc error: code = NotFound desc = could not find container \"077daf174c0dd94358e6e1b6b2f7afed5d6dfc274426435f5713cefc0543e0bd\": container with ID starting with 077daf174c0dd94358e6e1b6b2f7afed5d6dfc274426435f5713cefc0543e0bd not found: ID does not exist" Nov 02 22:56:24.687577 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:24.687520 5727 scope.go:122] "RemoveContainer" containerID="1658f55c0ad060b2eafb4957b547711ef2425e54b23c7088f56c4a3fc0815e69" Nov 02 22:56:24.696229 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:24.696206 5727 scope.go:122] "RemoveContainer" containerID="1658f55c0ad060b2eafb4957b547711ef2425e54b23c7088f56c4a3fc0815e69" Nov 02 22:56:24.696504 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:56:24.696484 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1658f55c0ad060b2eafb4957b547711ef2425e54b23c7088f56c4a3fc0815e69\": container with ID starting with 1658f55c0ad060b2eafb4957b547711ef2425e54b23c7088f56c4a3fc0815e69 not found: ID does not exist" containerID="1658f55c0ad060b2eafb4957b547711ef2425e54b23c7088f56c4a3fc0815e69" Nov 02 22:56:24.696574 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:24.696515 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1658f55c0ad060b2eafb4957b547711ef2425e54b23c7088f56c4a3fc0815e69"} err="failed to get container status \"1658f55c0ad060b2eafb4957b547711ef2425e54b23c7088f56c4a3fc0815e69\": rpc error: code = NotFound desc = could not find container \"1658f55c0ad060b2eafb4957b547711ef2425e54b23c7088f56c4a3fc0815e69\": container with ID starting with 1658f55c0ad060b2eafb4957b547711ef2425e54b23c7088f56c4a3fc0815e69 not found: ID does not exist" Nov 02 22:56:24.803065 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:24.803028 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-gcxrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Nov 02 22:56:24.803274 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:24.803088 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" podUID="c4c2f4f5-743d-4e9d-ae63-1769c521b4ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Nov 02 22:56:24.831253 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:24.831220 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5959c88d68-gdn7l"] Nov 02 22:56:24.835406 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:24.835383 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5959c88d68-gdn7l"] Nov 02 22:56:24.943654 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:24.943563 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-4bn4s"] Nov 02 22:56:24.945429 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:24.945397 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-4bn4s"] Nov 02 22:56:25.803262 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:25.803230 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-gcxrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Nov 02 22:56:25.803672 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:25.803283 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" podUID="c4c2f4f5-743d-4e9d-ae63-1769c521b4ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Nov 02 22:56:25.882600 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:25.882568 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="add45684-e452-47db-ae10-dc3a74827fd3" path="/var/lib/kubelet/pods/add45684-e452-47db-ae10-dc3a74827fd3/volumes" Nov 02 22:56:25.882890 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:25.882877 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6d5dfd6-5a95-4dee-97fe-209ffe1ed0e8" path="/var/lib/kubelet/pods/b6d5dfd6-5a95-4dee-97fe-209ffe1ed0e8/volumes" Nov 02 22:56:26.807039 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:26.807006 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:56:26.808288 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:26.808267 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" Nov 02 22:56:27.845011 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:56:27.844968 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 22:56:37.512245 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:37.512210 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmxhx" Nov 02 22:56:38.520232 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:56:38.520204 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-qprz9" Nov 02 22:56:57.844931 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:56:57.844893 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 22:57:27.845341 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:57:27.845301 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 22:57:57.845380 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:57:57.845342 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 22:58:00.984381 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:00.984343 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-vmxhx"] Nov 02 22:58:00.985326 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:00.984578 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmxhx" podUID="724a86df-d3a5-412d-afb9-becaa9cfd2cd" containerName="manager" containerID="cri-o://a76f50a037e4405d3940002a5034562f66511a1cbf8339327a16f6ab3aad31a6" gracePeriod=30 Nov 02 22:58:01.195959 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:01.195920 5727 generic.go:356] "Generic (PLEG): container finished" podID="724a86df-d3a5-412d-afb9-becaa9cfd2cd" containerID="a76f50a037e4405d3940002a5034562f66511a1cbf8339327a16f6ab3aad31a6" exitCode=0 Nov 02 22:58:01.196152 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:01.195985 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmxhx" event={"ID":"724a86df-d3a5-412d-afb9-becaa9cfd2cd","Type":"ContainerDied","Data":"a76f50a037e4405d3940002a5034562f66511a1cbf8339327a16f6ab3aad31a6"} Nov 02 22:58:01.237529 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:01.237475 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmxhx" Nov 02 22:58:01.264747 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:01.264719 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/724a86df-d3a5-412d-afb9-becaa9cfd2cd-cert\" (UniqueName: \"kubernetes.io/secret/724a86df-d3a5-412d-afb9-becaa9cfd2cd-cert\") pod \"724a86df-d3a5-412d-afb9-becaa9cfd2cd\" (UID: \"724a86df-d3a5-412d-afb9-becaa9cfd2cd\") " Nov 02 22:58:01.264875 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:01.264778 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/724a86df-d3a5-412d-afb9-becaa9cfd2cd-kube-api-access-bqdbx\" (UniqueName: \"kubernetes.io/projected/724a86df-d3a5-412d-afb9-becaa9cfd2cd-kube-api-access-bqdbx\") pod \"724a86df-d3a5-412d-afb9-becaa9cfd2cd\" (UID: \"724a86df-d3a5-412d-afb9-becaa9cfd2cd\") " Nov 02 22:58:01.267052 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:01.267018 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/724a86df-d3a5-412d-afb9-becaa9cfd2cd-cert" pod "724a86df-d3a5-412d-afb9-becaa9cfd2cd" (UID: "724a86df-d3a5-412d-afb9-becaa9cfd2cd"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:58:01.267167 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:01.267070 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/724a86df-d3a5-412d-afb9-becaa9cfd2cd-kube-api-access-bqdbx" pod "724a86df-d3a5-412d-afb9-becaa9cfd2cd" (UID: "724a86df-d3a5-412d-afb9-becaa9cfd2cd"). InnerVolumeSpecName "kube-api-access-bqdbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:58:01.365411 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:01.365382 5727 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/724a86df-d3a5-412d-afb9-becaa9cfd2cd-cert\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:58:01.365411 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:01.365407 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqdbx\" (UniqueName: \"kubernetes.io/projected/724a86df-d3a5-412d-afb9-becaa9cfd2cd-kube-api-access-bqdbx\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:58:02.163635 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:02.163550 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-rpfkq"] Nov 02 22:58:02.164152 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:02.164115 5727 cpu_manager.go:414] "containerMap: removing container" podUID="724a86df-d3a5-412d-afb9-becaa9cfd2cd" containerName="manager" Nov 02 22:58:02.164243 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:02.164155 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="724a86df-d3a5-412d-afb9-becaa9cfd2cd" containerName="manager" Nov 02 22:58:02.164243 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:02.164196 5727 cpu_manager.go:414] "containerMap: removing container" podUID="b6d5dfd6-5a95-4dee-97fe-209ffe1ed0e8" containerName="authorino" Nov 02 22:58:02.164243 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:02.164204 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6d5dfd6-5a95-4dee-97fe-209ffe1ed0e8" containerName="authorino" Nov 02 22:58:02.164243 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:02.164237 5727 cpu_manager.go:414] "containerMap: removing container" podUID="add45684-e452-47db-ae10-dc3a74827fd3" containerName="authorino" Nov 02 22:58:02.164439 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:02.164245 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="add45684-e452-47db-ae10-dc3a74827fd3" containerName="authorino" Nov 02 22:58:02.164439 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:02.164348 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="add45684-e452-47db-ae10-dc3a74827fd3" containerName="authorino" Nov 02 22:58:02.164439 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:02.164363 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="724a86df-d3a5-412d-afb9-becaa9cfd2cd" containerName="manager" Nov 02 22:58:02.164439 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:02.164377 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6d5dfd6-5a95-4dee-97fe-209ffe1ed0e8" containerName="authorino" Nov 02 22:58:02.166424 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:02.166405 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-rpfkq" Nov 02 22:58:02.171118 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:02.171098 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f30a1bf-8dc7-4072-8564-e8124bed9363-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-rpfkq\" (UID: \"7f30a1bf-8dc7-4072-8564-e8124bed9363\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-rpfkq" Nov 02 22:58:02.171332 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:02.171309 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68r7k\" (UniqueName: \"kubernetes.io/projected/7f30a1bf-8dc7-4072-8564-e8124bed9363-kube-api-access-68r7k\") pod \"ai-gateway-controller-5d8dddb676-rpfkq\" (UID: \"7f30a1bf-8dc7-4072-8564-e8124bed9363\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-rpfkq" Nov 02 22:58:02.203727 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:02.203689 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmxhx" event={"ID":"724a86df-d3a5-412d-afb9-becaa9cfd2cd","Type":"ContainerDied","Data":"c5b3861dbc288837fa523601cfe2e8952924fc5a4b3d583e8b9373f48a19bb64"} Nov 02 22:58:02.203901 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:02.203767 5727 scope.go:122] "RemoveContainer" containerID="a76f50a037e4405d3940002a5034562f66511a1cbf8339327a16f6ab3aad31a6" Nov 02 22:58:02.203901 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:02.203769 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmxhx" Nov 02 22:58:02.219422 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:02.219404 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 02 22:58:02.219507 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:02.219484 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 02 22:58:02.219674 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:02.219658 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-qns6m\"" Nov 02 22:58:02.270752 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:02.270724 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-rpfkq"] Nov 02 22:58:02.272058 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:02.272036 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68r7k\" (UniqueName: \"kubernetes.io/projected/7f30a1bf-8dc7-4072-8564-e8124bed9363-kube-api-access-68r7k\") pod \"ai-gateway-controller-5d8dddb676-rpfkq\" (UID: \"7f30a1bf-8dc7-4072-8564-e8124bed9363\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-rpfkq" Nov 02 22:58:02.272167 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:02.272085 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f30a1bf-8dc7-4072-8564-e8124bed9363-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-rpfkq\" (UID: \"7f30a1bf-8dc7-4072-8564-e8124bed9363\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-rpfkq" Nov 02 22:58:02.272246 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:58:02.272226 5727 secret.go:189] Couldn't get secret opendatahub/ai-gateway-controller-metrics-tls: secret "ai-gateway-controller-metrics-tls" not found Nov 02 22:58:02.272309 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:58:02.272298 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f30a1bf-8dc7-4072-8564-e8124bed9363-metrics-certs podName:7f30a1bf-8dc7-4072-8564-e8124bed9363 nodeName:}" failed. No retries permitted until 2026-11-02 22:58:02.772278535 +0000 UTC m=+569.010776190 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f30a1bf-8dc7-4072-8564-e8124bed9363-metrics-certs") pod "ai-gateway-controller-5d8dddb676-rpfkq" (UID: "7f30a1bf-8dc7-4072-8564-e8124bed9363") : secret "ai-gateway-controller-metrics-tls" not found Nov 02 22:58:02.334842 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:02.334814 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68r7k\" (UniqueName: \"kubernetes.io/projected/7f30a1bf-8dc7-4072-8564-e8124bed9363-kube-api-access-68r7k\") pod \"ai-gateway-controller-5d8dddb676-rpfkq\" (UID: \"7f30a1bf-8dc7-4072-8564-e8124bed9363\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-rpfkq" Nov 02 22:58:02.392640 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:02.392599 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-vmxhx"] Nov 02 22:58:02.398781 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:02.398755 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-vmxhx"] Nov 02 22:58:02.777581 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:02.777550 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f30a1bf-8dc7-4072-8564-e8124bed9363-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-rpfkq\" (UID: \"7f30a1bf-8dc7-4072-8564-e8124bed9363\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-rpfkq" Nov 02 22:58:02.780295 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:02.780264 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f30a1bf-8dc7-4072-8564-e8124bed9363-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-rpfkq\" (UID: \"7f30a1bf-8dc7-4072-8564-e8124bed9363\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-rpfkq" Nov 02 22:58:03.078557 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:03.078528 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-rpfkq" Nov 02 22:58:03.448310 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:03.448284 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-rpfkq"] Nov 02 22:58:03.450710 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:58:03.450677 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f30a1bf_8dc7_4072_8564_e8124bed9363.slice/crio-ac7ce871ce333f35a47a10e84260277412ec7ee40ce8d0fd2aed5bf1acd6b8ec WatchSource:0}: Error finding container ac7ce871ce333f35a47a10e84260277412ec7ee40ce8d0fd2aed5bf1acd6b8ec: Status 404 returned error can't find the container with id ac7ce871ce333f35a47a10e84260277412ec7ee40ce8d0fd2aed5bf1acd6b8ec Nov 02 22:58:03.881942 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:03.881899 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="724a86df-d3a5-412d-afb9-becaa9cfd2cd" path="/var/lib/kubelet/pods/724a86df-d3a5-412d-afb9-becaa9cfd2cd/volumes" Nov 02 22:58:04.216065 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:04.215975 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-rpfkq" event={"ID":"7f30a1bf-8dc7-4072-8564-e8124bed9363","Type":"ContainerStarted","Data":"ac7ce871ce333f35a47a10e84260277412ec7ee40ce8d0fd2aed5bf1acd6b8ec"} Nov 02 22:58:07.236808 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:07.236770 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-rpfkq" event={"ID":"7f30a1bf-8dc7-4072-8564-e8124bed9363","Type":"ContainerStarted","Data":"dc9102b9e960a42da611a2b972d46bdd3e240f09d0bf8d966df67a277300af39"} Nov 02 22:58:07.237261 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:07.236902 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-rpfkq" Nov 02 22:58:07.352150 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:07.352073 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d8dddb676-rpfkq" podStartSLOduration=2.485427313 podStartE2EDuration="5.352056932s" totalImagesPullingTime="2.866629619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:58:07.352032415 +0000 UTC m=+573.590530091" watchObservedRunningTime="2026-11-02 22:58:07.352056932 +0000 UTC m=+573.590554598" Nov 02 22:58:10.627294 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:10.627264 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-cdflj"] Nov 02 22:58:10.629795 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:10.629778 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cdflj" Nov 02 22:58:10.647267 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:10.647244 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjptc\" (UniqueName: \"kubernetes.io/projected/c0c34a06-6f76-4573-9cc8-7ba13cc79280-kube-api-access-mjptc\") pod \"payload-pre-processing-7b98bb678c-cdflj\" (UID: \"c0c34a06-6f76-4573-9cc8-7ba13cc79280\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-cdflj" Nov 02 22:58:10.647368 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:10.647279 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0c34a06-6f76-4573-9cc8-7ba13cc79280-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-cdflj\" (UID: \"c0c34a06-6f76-4573-9cc8-7ba13cc79280\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-cdflj" Nov 02 22:58:10.731200 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:10.731173 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-kklzm\"" Nov 02 22:58:10.731200 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:10.731174 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 02 22:58:10.733413 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:10.733391 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-cdflj"] Nov 02 22:58:10.747762 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:10.747737 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjptc\" (UniqueName: \"kubernetes.io/projected/c0c34a06-6f76-4573-9cc8-7ba13cc79280-kube-api-access-mjptc\") pod \"payload-pre-processing-7b98bb678c-cdflj\" (UID: \"c0c34a06-6f76-4573-9cc8-7ba13cc79280\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-cdflj" Nov 02 22:58:10.747891 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:10.747775 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0c34a06-6f76-4573-9cc8-7ba13cc79280-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-cdflj\" (UID: \"c0c34a06-6f76-4573-9cc8-7ba13cc79280\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-cdflj" Nov 02 22:58:10.748484 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:10.748464 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0c34a06-6f76-4573-9cc8-7ba13cc79280-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-cdflj\" (UID: \"c0c34a06-6f76-4573-9cc8-7ba13cc79280\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-cdflj" Nov 02 22:58:10.784564 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:10.784525 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-279zt"] Nov 02 22:58:10.788425 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:10.788400 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-279zt" Nov 02 22:58:10.811576 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:10.811550 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjptc\" (UniqueName: \"kubernetes.io/projected/c0c34a06-6f76-4573-9cc8-7ba13cc79280-kube-api-access-mjptc\") pod \"payload-pre-processing-7b98bb678c-cdflj\" (UID: \"c0c34a06-6f76-4573-9cc8-7ba13cc79280\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-cdflj" Nov 02 22:58:10.832448 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:10.832420 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-279zt"] Nov 02 22:58:10.848373 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:10.848340 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qvmf\" (UniqueName: \"kubernetes.io/projected/f36665e4-38c7-4d93-b047-306185b1a17f-kube-api-access-9qvmf\") pod \"payload-processing-55fbbbf9dc-279zt\" (UID: \"f36665e4-38c7-4d93-b047-306185b1a17f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-279zt" Nov 02 22:58:10.848568 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:10.848395 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f36665e4-38c7-4d93-b047-306185b1a17f-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-279zt\" (UID: \"f36665e4-38c7-4d93-b047-306185b1a17f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-279zt" Nov 02 22:58:10.942345 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:10.942270 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cdflj" Nov 02 22:58:10.949590 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:10.949564 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qvmf\" (UniqueName: \"kubernetes.io/projected/f36665e4-38c7-4d93-b047-306185b1a17f-kube-api-access-9qvmf\") pod \"payload-processing-55fbbbf9dc-279zt\" (UID: \"f36665e4-38c7-4d93-b047-306185b1a17f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-279zt" Nov 02 22:58:10.949709 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:10.949598 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f36665e4-38c7-4d93-b047-306185b1a17f-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-279zt\" (UID: \"f36665e4-38c7-4d93-b047-306185b1a17f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-279zt" Nov 02 22:58:10.950187 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:10.950168 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f36665e4-38c7-4d93-b047-306185b1a17f-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-279zt\" (UID: \"f36665e4-38c7-4d93-b047-306185b1a17f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-279zt" Nov 02 22:58:11.010637 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:11.010577 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qvmf\" (UniqueName: \"kubernetes.io/projected/f36665e4-38c7-4d93-b047-306185b1a17f-kube-api-access-9qvmf\") pod \"payload-processing-55fbbbf9dc-279zt\" (UID: \"f36665e4-38c7-4d93-b047-306185b1a17f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-279zt" Nov 02 22:58:11.102030 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:11.102003 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-279zt" Nov 02 22:58:11.319641 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:11.319617 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-cdflj"] Nov 02 22:58:11.322024 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:58:11.321998 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0c34a06_6f76_4573_9cc8_7ba13cc79280.slice/crio-46950a0261aef0bb4e0e10868f31e3946b2753a9db89b025aae343c47bfee4f5 WatchSource:0}: Error finding container 46950a0261aef0bb4e0e10868f31e3946b2753a9db89b025aae343c47bfee4f5: Status 404 returned error can't find the container with id 46950a0261aef0bb4e0e10868f31e3946b2753a9db89b025aae343c47bfee4f5 Nov 02 22:58:11.471600 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:11.471576 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-279zt"] Nov 02 22:58:11.473667 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:58:11.473638 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf36665e4_38c7_4d93_b047_306185b1a17f.slice/crio-8026f4378626f6290c45434097f10ad504476324e776d23e08d6e4104f4374cd WatchSource:0}: Error finding container 8026f4378626f6290c45434097f10ad504476324e776d23e08d6e4104f4374cd: Status 404 returned error can't find the container with id 8026f4378626f6290c45434097f10ad504476324e776d23e08d6e4104f4374cd Nov 02 22:58:12.264324 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:12.264289 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-279zt" event={"ID":"f36665e4-38c7-4d93-b047-306185b1a17f","Type":"ContainerStarted","Data":"8026f4378626f6290c45434097f10ad504476324e776d23e08d6e4104f4374cd"} Nov 02 22:58:12.265560 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:12.265539 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cdflj" event={"ID":"c0c34a06-6f76-4573-9cc8-7ba13cc79280","Type":"ContainerStarted","Data":"46950a0261aef0bb4e0e10868f31e3946b2753a9db89b025aae343c47bfee4f5"} Nov 02 22:58:14.280826 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:14.280787 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-279zt" event={"ID":"f36665e4-38c7-4d93-b047-306185b1a17f","Type":"ContainerStarted","Data":"76b3e32e5099d40d1029f75312d4017749e2d499b1e86121d04fcffa9cc12973"} Nov 02 22:58:14.281240 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:14.280913 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-279zt" Nov 02 22:58:14.282426 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:14.282402 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cdflj" event={"ID":"c0c34a06-6f76-4573-9cc8-7ba13cc79280","Type":"ContainerStarted","Data":"0739013299d944ad22f4ad12b21e669e27573314fce73b181ec8db3eafb6f853"} Nov 02 22:58:14.282549 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:14.282447 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cdflj" Nov 02 22:58:14.396019 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:14.395933 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-279zt" podStartSLOduration=1.7493016 podStartE2EDuration="4.395917879s" totalImagesPullingTime="2.646616279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:58:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:58:14.395826553 +0000 UTC m=+580.634324233" watchObservedRunningTime="2026-11-02 22:58:14.395917879 +0000 UTC m=+580.634415542" Nov 02 22:58:14.510805 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:14.510754 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cdflj" podStartSLOduration=1.690224541 podStartE2EDuration="4.51073994s" totalImagesPullingTime="2.820515399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:58:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:58:14.510447294 +0000 UTC m=+580.748944969" watchObservedRunningTime="2026-11-02 22:58:14.51073994 +0000 UTC m=+580.749237602" Nov 02 22:58:18.244871 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:18.244837 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-rpfkq" Nov 02 22:58:18.609419 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:18.609388 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7fcfb78cb5-m2vnt"] Nov 02 22:58:18.612883 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:18.612866 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7fcfb78cb5-m2vnt" Nov 02 22:58:18.670526 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:18.670501 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7fcfb78cb5-m2vnt"] Nov 02 22:58:18.718343 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:18.718317 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hcpq\" (UniqueName: \"kubernetes.io/projected/b2101b43-6e2c-42d8-8d60-d44faa6433f1-kube-api-access-9hcpq\") pod \"ai-gateway-controller-7fcfb78cb5-m2vnt\" (UID: \"b2101b43-6e2c-42d8-8d60-d44faa6433f1\") " pod="opendatahub/ai-gateway-controller-7fcfb78cb5-m2vnt" Nov 02 22:58:18.718469 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:18.718351 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2101b43-6e2c-42d8-8d60-d44faa6433f1-metrics-certs\") pod \"ai-gateway-controller-7fcfb78cb5-m2vnt\" (UID: \"b2101b43-6e2c-42d8-8d60-d44faa6433f1\") " pod="opendatahub/ai-gateway-controller-7fcfb78cb5-m2vnt" Nov 02 22:58:18.819817 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:18.819789 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hcpq\" (UniqueName: \"kubernetes.io/projected/b2101b43-6e2c-42d8-8d60-d44faa6433f1-kube-api-access-9hcpq\") pod \"ai-gateway-controller-7fcfb78cb5-m2vnt\" (UID: \"b2101b43-6e2c-42d8-8d60-d44faa6433f1\") " pod="opendatahub/ai-gateway-controller-7fcfb78cb5-m2vnt" Nov 02 22:58:18.819965 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:18.819829 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2101b43-6e2c-42d8-8d60-d44faa6433f1-metrics-certs\") pod \"ai-gateway-controller-7fcfb78cb5-m2vnt\" (UID: \"b2101b43-6e2c-42d8-8d60-d44faa6433f1\") " pod="opendatahub/ai-gateway-controller-7fcfb78cb5-m2vnt" Nov 02 22:58:18.822361 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:18.822334 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2101b43-6e2c-42d8-8d60-d44faa6433f1-metrics-certs\") pod \"ai-gateway-controller-7fcfb78cb5-m2vnt\" (UID: \"b2101b43-6e2c-42d8-8d60-d44faa6433f1\") " pod="opendatahub/ai-gateway-controller-7fcfb78cb5-m2vnt" Nov 02 22:58:18.880855 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:18.880777 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hcpq\" (UniqueName: \"kubernetes.io/projected/b2101b43-6e2c-42d8-8d60-d44faa6433f1-kube-api-access-9hcpq\") pod \"ai-gateway-controller-7fcfb78cb5-m2vnt\" (UID: \"b2101b43-6e2c-42d8-8d60-d44faa6433f1\") " pod="opendatahub/ai-gateway-controller-7fcfb78cb5-m2vnt" Nov 02 22:58:18.925922 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:18.925891 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7fcfb78cb5-m2vnt" Nov 02 22:58:19.299599 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:19.299577 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7fcfb78cb5-m2vnt"] Nov 02 22:58:19.301825 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:58:19.301799 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2101b43_6e2c_42d8_8d60_d44faa6433f1.slice/crio-2f12610cd9adebb9216ca3a499ae482e0cb46d65e736e33e69feadfc688ca916 WatchSource:0}: Error finding container 2f12610cd9adebb9216ca3a499ae482e0cb46d65e736e33e69feadfc688ca916: Status 404 returned error can't find the container with id 2f12610cd9adebb9216ca3a499ae482e0cb46d65e736e33e69feadfc688ca916 Nov 02 22:58:19.312303 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:19.312275 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7fcfb78cb5-m2vnt" event={"ID":"b2101b43-6e2c-42d8-8d60-d44faa6433f1","Type":"ContainerStarted","Data":"2f12610cd9adebb9216ca3a499ae482e0cb46d65e736e33e69feadfc688ca916"} Nov 02 22:58:20.319892 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:20.319857 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7fcfb78cb5-m2vnt" event={"ID":"b2101b43-6e2c-42d8-8d60-d44faa6433f1","Type":"ContainerStarted","Data":"28a7bac1299a749de3f02e5d8e99638d57d8ed994fa0cfc5d58d13ce299dfd2f"} Nov 02 22:58:20.320302 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:20.319922 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7fcfb78cb5-m2vnt" Nov 02 22:58:20.435669 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:20.435567 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7fcfb78cb5-m2vnt" podStartSLOduration=1.5924871299999999 podStartE2EDuration="2.4355526s" totalImagesPullingTime="843.06547ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:58:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:58:20.434520886 +0000 UTC m=+586.673018548" watchObservedRunningTime="2026-11-02 22:58:20.4355526 +0000 UTC m=+586.674050262" Nov 02 22:58:25.291020 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:25.290994 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-279zt" Nov 02 22:58:25.291430 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:25.291054 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cdflj" Nov 02 22:58:27.844480 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:58:27.844438 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 22:58:31.328673 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:31.328640 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7fcfb78cb5-m2vnt" Nov 02 22:58:31.496527 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:31.496491 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-rpfkq"] Nov 02 22:58:31.496808 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:31.496781 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d8dddb676-rpfkq" podUID="7f30a1bf-8dc7-4072-8564-e8124bed9363" containerName="manager" containerID="cri-o://dc9102b9e960a42da611a2b972d46bdd3e240f09d0bf8d966df67a277300af39" gracePeriod=30 Nov 02 22:58:31.757938 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:31.757913 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-rpfkq" Nov 02 22:58:31.843430 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:31.843399 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f30a1bf-8dc7-4072-8564-e8124bed9363-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f30a1bf-8dc7-4072-8564-e8124bed9363-metrics-certs\") pod \"7f30a1bf-8dc7-4072-8564-e8124bed9363\" (UID: \"7f30a1bf-8dc7-4072-8564-e8124bed9363\") " Nov 02 22:58:31.843588 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:31.843536 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f30a1bf-8dc7-4072-8564-e8124bed9363-kube-api-access-68r7k\" (UniqueName: \"kubernetes.io/projected/7f30a1bf-8dc7-4072-8564-e8124bed9363-kube-api-access-68r7k\") pod \"7f30a1bf-8dc7-4072-8564-e8124bed9363\" (UID: \"7f30a1bf-8dc7-4072-8564-e8124bed9363\") " Nov 02 22:58:31.845645 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:31.845617 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f30a1bf-8dc7-4072-8564-e8124bed9363-metrics-certs" pod "7f30a1bf-8dc7-4072-8564-e8124bed9363" (UID: "7f30a1bf-8dc7-4072-8564-e8124bed9363"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:58:31.845747 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:31.845660 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f30a1bf-8dc7-4072-8564-e8124bed9363-kube-api-access-68r7k" pod "7f30a1bf-8dc7-4072-8564-e8124bed9363" (UID: "7f30a1bf-8dc7-4072-8564-e8124bed9363"). InnerVolumeSpecName "kube-api-access-68r7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:58:31.944269 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:31.944230 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68r7k\" (UniqueName: \"kubernetes.io/projected/7f30a1bf-8dc7-4072-8564-e8124bed9363-kube-api-access-68r7k\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:58:31.944269 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:31.944264 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f30a1bf-8dc7-4072-8564-e8124bed9363-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:58:32.385721 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:32.385677 5727 generic.go:356] "Generic (PLEG): container finished" podID="7f30a1bf-8dc7-4072-8564-e8124bed9363" containerID="dc9102b9e960a42da611a2b972d46bdd3e240f09d0bf8d966df67a277300af39" exitCode=0 Nov 02 22:58:32.386182 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:32.385728 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-rpfkq" Nov 02 22:58:32.386182 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:32.385759 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-rpfkq" event={"ID":"7f30a1bf-8dc7-4072-8564-e8124bed9363","Type":"ContainerDied","Data":"dc9102b9e960a42da611a2b972d46bdd3e240f09d0bf8d966df67a277300af39"} Nov 02 22:58:32.386182 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:32.385818 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-rpfkq" event={"ID":"7f30a1bf-8dc7-4072-8564-e8124bed9363","Type":"ContainerDied","Data":"ac7ce871ce333f35a47a10e84260277412ec7ee40ce8d0fd2aed5bf1acd6b8ec"} Nov 02 22:58:32.386182 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:32.385842 5727 scope.go:122] "RemoveContainer" containerID="dc9102b9e960a42da611a2b972d46bdd3e240f09d0bf8d966df67a277300af39" Nov 02 22:58:32.395714 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:32.395698 5727 scope.go:122] "RemoveContainer" containerID="dc9102b9e960a42da611a2b972d46bdd3e240f09d0bf8d966df67a277300af39" Nov 02 22:58:32.395988 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:58:32.395971 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc9102b9e960a42da611a2b972d46bdd3e240f09d0bf8d966df67a277300af39\": container with ID starting with dc9102b9e960a42da611a2b972d46bdd3e240f09d0bf8d966df67a277300af39 not found: ID does not exist" containerID="dc9102b9e960a42da611a2b972d46bdd3e240f09d0bf8d966df67a277300af39" Nov 02 22:58:32.396047 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:32.396000 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc9102b9e960a42da611a2b972d46bdd3e240f09d0bf8d966df67a277300af39"} err="failed to get container status \"dc9102b9e960a42da611a2b972d46bdd3e240f09d0bf8d966df67a277300af39\": rpc error: code = NotFound desc = could not find container \"dc9102b9e960a42da611a2b972d46bdd3e240f09d0bf8d966df67a277300af39\": container with ID starting with dc9102b9e960a42da611a2b972d46bdd3e240f09d0bf8d966df67a277300af39 not found: ID does not exist" Nov 02 22:58:32.553761 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:32.553717 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-rpfkq"] Nov 02 22:58:32.603407 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:32.603374 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-rpfkq"] Nov 02 22:58:33.882436 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:33.882404 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f30a1bf-8dc7-4072-8564-e8124bed9363" path="/var/lib/kubelet/pods/7f30a1bf-8dc7-4072-8564-e8124bed9363/volumes" Nov 02 22:58:34.168638 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:34.168562 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 02 22:58:34.184880 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:34.184843 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 22:58:34.194205 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:34.194181 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 02 22:58:34.208229 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:34.208203 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 22:58:41.090960 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:41.090929 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9"] Nov 02 22:58:41.091361 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:41.091348 5727 cpu_manager.go:414] "containerMap: removing container" podUID="7f30a1bf-8dc7-4072-8564-e8124bed9363" containerName="manager" Nov 02 22:58:41.091413 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:41.091362 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f30a1bf-8dc7-4072-8564-e8124bed9363" containerName="manager" Nov 02 22:58:41.091464 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:41.091456 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f30a1bf-8dc7-4072-8564-e8124bed9363" containerName="manager" Nov 02 22:58:41.096506 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:41.096484 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9" Nov 02 22:58:41.200416 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:41.200385 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 22:58:41.200416 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:41.200395 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 02 22:58:41.200416 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:41.200414 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 22:58:41.200711 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:41.200466 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tzw2x\"" Nov 02 22:58:41.202644 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:41.202622 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9"] Nov 02 22:58:41.230974 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:41.230899 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwpwd\" (UniqueName: \"kubernetes.io/projected/00897aef-01f7-4dcf-91df-46eca1fb734b-kube-api-access-vwpwd\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9\" (UID: \"00897aef-01f7-4dcf-91df-46eca1fb734b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9" Nov 02 22:58:41.230974 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:41.230942 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00897aef-01f7-4dcf-91df-46eca1fb734b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9\" (UID: \"00897aef-01f7-4dcf-91df-46eca1fb734b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9" Nov 02 22:58:41.231247 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:41.231028 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00897aef-01f7-4dcf-91df-46eca1fb734b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9\" (UID: \"00897aef-01f7-4dcf-91df-46eca1fb734b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9" Nov 02 22:58:41.231247 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:41.231209 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00897aef-01f7-4dcf-91df-46eca1fb734b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9\" (UID: \"00897aef-01f7-4dcf-91df-46eca1fb734b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9" Nov 02 22:58:41.231335 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:41.231263 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00897aef-01f7-4dcf-91df-46eca1fb734b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9\" (UID: \"00897aef-01f7-4dcf-91df-46eca1fb734b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9" Nov 02 22:58:41.231335 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:41.231293 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00897aef-01f7-4dcf-91df-46eca1fb734b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9\" (UID: \"00897aef-01f7-4dcf-91df-46eca1fb734b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9" Nov 02 22:58:41.332420 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:41.332390 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00897aef-01f7-4dcf-91df-46eca1fb734b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9\" (UID: \"00897aef-01f7-4dcf-91df-46eca1fb734b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9" Nov 02 22:58:41.332601 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:41.332461 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00897aef-01f7-4dcf-91df-46eca1fb734b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9\" (UID: \"00897aef-01f7-4dcf-91df-46eca1fb734b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9" Nov 02 22:58:41.332601 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:41.332485 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00897aef-01f7-4dcf-91df-46eca1fb734b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9\" (UID: \"00897aef-01f7-4dcf-91df-46eca1fb734b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9" Nov 02 22:58:41.332601 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:41.332502 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00897aef-01f7-4dcf-91df-46eca1fb734b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9\" (UID: \"00897aef-01f7-4dcf-91df-46eca1fb734b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9" Nov 02 22:58:41.332601 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:41.332524 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwpwd\" (UniqueName: \"kubernetes.io/projected/00897aef-01f7-4dcf-91df-46eca1fb734b-kube-api-access-vwpwd\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9\" (UID: \"00897aef-01f7-4dcf-91df-46eca1fb734b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9" Nov 02 22:58:41.332601 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:41.332553 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00897aef-01f7-4dcf-91df-46eca1fb734b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9\" (UID: \"00897aef-01f7-4dcf-91df-46eca1fb734b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9" Nov 02 22:58:41.332907 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:41.332882 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00897aef-01f7-4dcf-91df-46eca1fb734b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9\" (UID: \"00897aef-01f7-4dcf-91df-46eca1fb734b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9" Nov 02 22:58:41.333012 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:41.332932 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00897aef-01f7-4dcf-91df-46eca1fb734b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9\" (UID: \"00897aef-01f7-4dcf-91df-46eca1fb734b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9" Nov 02 22:58:41.333012 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:41.332960 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00897aef-01f7-4dcf-91df-46eca1fb734b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9\" (UID: \"00897aef-01f7-4dcf-91df-46eca1fb734b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9" Nov 02 22:58:41.334719 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:41.334702 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00897aef-01f7-4dcf-91df-46eca1fb734b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9\" (UID: \"00897aef-01f7-4dcf-91df-46eca1fb734b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9" Nov 02 22:58:41.335038 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:41.335022 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00897aef-01f7-4dcf-91df-46eca1fb734b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9\" (UID: \"00897aef-01f7-4dcf-91df-46eca1fb734b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9" Nov 02 22:58:41.393374 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:41.393316 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwpwd\" (UniqueName: \"kubernetes.io/projected/00897aef-01f7-4dcf-91df-46eca1fb734b-kube-api-access-vwpwd\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9\" (UID: \"00897aef-01f7-4dcf-91df-46eca1fb734b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9" Nov 02 22:58:41.409186 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:41.409160 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9" Nov 02 22:58:41.785928 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:41.785896 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9"] Nov 02 22:58:41.788053 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:58:41.788009 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00897aef_01f7_4dcf_91df_46eca1fb734b.slice/crio-74b437cd6a003f01773aaa3eb1b9e6b20a4ff29c2895dc08ac3b982f191310e7 WatchSource:0}: Error finding container 74b437cd6a003f01773aaa3eb1b9e6b20a4ff29c2895dc08ac3b982f191310e7: Status 404 returned error can't find the container with id 74b437cd6a003f01773aaa3eb1b9e6b20a4ff29c2895dc08ac3b982f191310e7 Nov 02 22:58:42.444255 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:42.444217 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9" event={"ID":"00897aef-01f7-4dcf-91df-46eca1fb734b","Type":"ContainerStarted","Data":"74b437cd6a003f01773aaa3eb1b9e6b20a4ff29c2895dc08ac3b982f191310e7"} Nov 02 22:58:44.460079 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:44.460041 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9" event={"ID":"00897aef-01f7-4dcf-91df-46eca1fb734b","Type":"ContainerStarted","Data":"1e77ef1324d2e5d1b0ffdde9c7f14020cfadfff4fe40f22d629863b1d7b9d58a"} Nov 02 22:58:44.578187 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:44.578117 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9" podStartSLOduration=1.778377774 podStartE2EDuration="3.578104532s" totalImagesPullingTime="1.799726758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:58:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:58:44.57625175 +0000 UTC m=+610.814749425" watchObservedRunningTime="2026-11-02 22:58:44.578104532 +0000 UTC m=+610.816602194" Nov 02 22:58:45.410191 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:45.410154 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9" Nov 02 22:58:48.883586 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:48.883548 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-bj2k2"] Nov 02 22:58:48.890536 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:48.890514 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bj2k2" Nov 02 22:58:48.943627 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:48.943599 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 02 22:58:48.994681 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:48.994653 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-bj2k2"] Nov 02 22:58:49.000804 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:49.000785 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b33c63bc-b6b9-412d-bd69-b0e2c3ebf353-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bj2k2\" (UID: \"b33c63bc-b6b9-412d-bd69-b0e2c3ebf353\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bj2k2" Nov 02 22:58:49.000882 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:49.000812 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b33c63bc-b6b9-412d-bd69-b0e2c3ebf353-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bj2k2\" (UID: \"b33c63bc-b6b9-412d-bd69-b0e2c3ebf353\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bj2k2" Nov 02 22:58:49.000949 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:49.000934 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b33c63bc-b6b9-412d-bd69-b0e2c3ebf353-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bj2k2\" (UID: \"b33c63bc-b6b9-412d-bd69-b0e2c3ebf353\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bj2k2" Nov 02 22:58:49.001074 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:49.001058 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b33c63bc-b6b9-412d-bd69-b0e2c3ebf353-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bj2k2\" (UID: \"b33c63bc-b6b9-412d-bd69-b0e2c3ebf353\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bj2k2" Nov 02 22:58:49.001111 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:49.001090 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b33c63bc-b6b9-412d-bd69-b0e2c3ebf353-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bj2k2\" (UID: \"b33c63bc-b6b9-412d-bd69-b0e2c3ebf353\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bj2k2" Nov 02 22:58:49.001182 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:49.001115 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrts2\" (UniqueName: \"kubernetes.io/projected/b33c63bc-b6b9-412d-bd69-b0e2c3ebf353-kube-api-access-xrts2\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bj2k2\" (UID: \"b33c63bc-b6b9-412d-bd69-b0e2c3ebf353\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bj2k2" Nov 02 22:58:49.101681 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:49.101648 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b33c63bc-b6b9-412d-bd69-b0e2c3ebf353-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bj2k2\" (UID: \"b33c63bc-b6b9-412d-bd69-b0e2c3ebf353\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bj2k2" Nov 02 22:58:49.101681 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:49.101686 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b33c63bc-b6b9-412d-bd69-b0e2c3ebf353-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bj2k2\" (UID: \"b33c63bc-b6b9-412d-bd69-b0e2c3ebf353\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bj2k2" Nov 02 22:58:49.101927 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:49.101740 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b33c63bc-b6b9-412d-bd69-b0e2c3ebf353-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bj2k2\" (UID: \"b33c63bc-b6b9-412d-bd69-b0e2c3ebf353\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bj2k2" Nov 02 22:58:49.101927 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:49.101816 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b33c63bc-b6b9-412d-bd69-b0e2c3ebf353-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bj2k2\" (UID: \"b33c63bc-b6b9-412d-bd69-b0e2c3ebf353\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bj2k2" Nov 02 22:58:49.101927 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:49.101837 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b33c63bc-b6b9-412d-bd69-b0e2c3ebf353-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bj2k2\" (UID: \"b33c63bc-b6b9-412d-bd69-b0e2c3ebf353\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bj2k2" Nov 02 22:58:49.101927 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:49.101858 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrts2\" (UniqueName: \"kubernetes.io/projected/b33c63bc-b6b9-412d-bd69-b0e2c3ebf353-kube-api-access-xrts2\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bj2k2\" (UID: \"b33c63bc-b6b9-412d-bd69-b0e2c3ebf353\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bj2k2" Nov 02 22:58:49.102193 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:49.102169 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b33c63bc-b6b9-412d-bd69-b0e2c3ebf353-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bj2k2\" (UID: \"b33c63bc-b6b9-412d-bd69-b0e2c3ebf353\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bj2k2" Nov 02 22:58:49.102267 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:49.102199 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b33c63bc-b6b9-412d-bd69-b0e2c3ebf353-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bj2k2\" (UID: \"b33c63bc-b6b9-412d-bd69-b0e2c3ebf353\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bj2k2" Nov 02 22:58:49.102311 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:49.102267 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b33c63bc-b6b9-412d-bd69-b0e2c3ebf353-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bj2k2\" (UID: \"b33c63bc-b6b9-412d-bd69-b0e2c3ebf353\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bj2k2" Nov 02 22:58:49.104277 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:49.104254 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b33c63bc-b6b9-412d-bd69-b0e2c3ebf353-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bj2k2\" (UID: \"b33c63bc-b6b9-412d-bd69-b0e2c3ebf353\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bj2k2" Nov 02 22:58:49.104529 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:49.104512 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b33c63bc-b6b9-412d-bd69-b0e2c3ebf353-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bj2k2\" (UID: \"b33c63bc-b6b9-412d-bd69-b0e2c3ebf353\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bj2k2" Nov 02 22:58:49.162744 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:49.162669 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrts2\" (UniqueName: \"kubernetes.io/projected/b33c63bc-b6b9-412d-bd69-b0e2c3ebf353-kube-api-access-xrts2\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bj2k2\" (UID: \"b33c63bc-b6b9-412d-bd69-b0e2c3ebf353\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bj2k2" Nov 02 22:58:49.203413 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:49.203379 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bj2k2" Nov 02 22:58:49.581370 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:49.581349 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-bj2k2"] Nov 02 22:58:49.583371 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:58:49.583347 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33c63bc_b6b9_412d_bd69_b0e2c3ebf353.slice/crio-0ae1015e07eec4d7069caae93b02a8d9a146249260dcb18d97036e96507d0da8 WatchSource:0}: Error finding container 0ae1015e07eec4d7069caae93b02a8d9a146249260dcb18d97036e96507d0da8: Status 404 returned error can't find the container with id 0ae1015e07eec4d7069caae93b02a8d9a146249260dcb18d97036e96507d0da8 Nov 02 22:58:50.495921 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:50.495839 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bj2k2" event={"ID":"b33c63bc-b6b9-412d-bd69-b0e2c3ebf353","Type":"ContainerStarted","Data":"6883effea0e52e9cd9dac5aa12262803b71582a9c2a2c34a64ecd5662cda0a24"} Nov 02 22:58:50.495921 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:50.495885 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bj2k2" event={"ID":"b33c63bc-b6b9-412d-bd69-b0e2c3ebf353","Type":"ContainerStarted","Data":"0ae1015e07eec4d7069caae93b02a8d9a146249260dcb18d97036e96507d0da8"} Nov 02 22:58:50.612713 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:50.612655 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bj2k2" podStartSLOduration=2.025864828 podStartE2EDuration="2.612639975s" totalImagesPullingTime="586.775147ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:58:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:58:50.612247426 +0000 UTC m=+616.850745089" watchObservedRunningTime="2026-11-02 22:58:50.612639975 +0000 UTC m=+616.851137637" Nov 02 22:58:51.203648 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:51.203615 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bj2k2" Nov 02 22:58:51.409306 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:51.409270 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9" Nov 02 22:58:51.422650 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:51.422621 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9" Nov 02 22:58:51.513792 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:51.513710 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9" Nov 02 22:58:57.844321 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:58:57.844281 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 22:58:59.203603 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:59.203569 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bj2k2" Nov 02 22:58:59.216038 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:59.216018 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bj2k2" Nov 02 22:58:59.558703 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:58:59.558631 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bj2k2" Nov 02 22:59:06.431757 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:06.431724 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-b986v"] Nov 02 22:59:06.435588 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:06.435569 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-b986v" Nov 02 22:59:06.536890 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:06.536858 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 02 22:59:06.539314 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:06.539292 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-b986v"] Nov 02 22:59:06.570144 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:06.570084 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gzg2\" (UniqueName: \"kubernetes.io/projected/f674fe20-ea05-4246-a4ff-7cd1da321621-kube-api-access-5gzg2\") pod \"echo-simulator-kserve-64b6f8f96c-b986v\" (UID: \"f674fe20-ea05-4246-a4ff-7cd1da321621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b986v" Nov 02 22:59:06.570144 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:06.570143 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f674fe20-ea05-4246-a4ff-7cd1da321621-home\") pod \"echo-simulator-kserve-64b6f8f96c-b986v\" (UID: \"f674fe20-ea05-4246-a4ff-7cd1da321621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b986v" Nov 02 22:59:06.570474 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:06.570246 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f674fe20-ea05-4246-a4ff-7cd1da321621-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-b986v\" (UID: \"f674fe20-ea05-4246-a4ff-7cd1da321621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b986v" Nov 02 22:59:06.570474 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:06.570384 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f674fe20-ea05-4246-a4ff-7cd1da321621-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-b986v\" (UID: \"f674fe20-ea05-4246-a4ff-7cd1da321621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b986v" Nov 02 22:59:06.570474 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:06.570414 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f674fe20-ea05-4246-a4ff-7cd1da321621-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-b986v\" (UID: \"f674fe20-ea05-4246-a4ff-7cd1da321621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b986v" Nov 02 22:59:06.570613 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:06.570479 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f674fe20-ea05-4246-a4ff-7cd1da321621-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-b986v\" (UID: \"f674fe20-ea05-4246-a4ff-7cd1da321621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b986v" Nov 02 22:59:06.671579 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:06.671548 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gzg2\" (UniqueName: \"kubernetes.io/projected/f674fe20-ea05-4246-a4ff-7cd1da321621-kube-api-access-5gzg2\") pod \"echo-simulator-kserve-64b6f8f96c-b986v\" (UID: \"f674fe20-ea05-4246-a4ff-7cd1da321621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b986v" Nov 02 22:59:06.671579 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:06.671583 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f674fe20-ea05-4246-a4ff-7cd1da321621-home\") pod \"echo-simulator-kserve-64b6f8f96c-b986v\" (UID: \"f674fe20-ea05-4246-a4ff-7cd1da321621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b986v" Nov 02 22:59:06.671810 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:06.671614 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f674fe20-ea05-4246-a4ff-7cd1da321621-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-b986v\" (UID: \"f674fe20-ea05-4246-a4ff-7cd1da321621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b986v" Nov 02 22:59:06.671810 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:06.671649 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f674fe20-ea05-4246-a4ff-7cd1da321621-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-b986v\" (UID: \"f674fe20-ea05-4246-a4ff-7cd1da321621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b986v" Nov 02 22:59:06.671810 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:06.671663 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f674fe20-ea05-4246-a4ff-7cd1da321621-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-b986v\" (UID: \"f674fe20-ea05-4246-a4ff-7cd1da321621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b986v" Nov 02 22:59:06.671810 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:06.671702 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f674fe20-ea05-4246-a4ff-7cd1da321621-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-b986v\" (UID: \"f674fe20-ea05-4246-a4ff-7cd1da321621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b986v" Nov 02 22:59:06.672114 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:06.672082 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f674fe20-ea05-4246-a4ff-7cd1da321621-home\") pod \"echo-simulator-kserve-64b6f8f96c-b986v\" (UID: \"f674fe20-ea05-4246-a4ff-7cd1da321621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b986v" Nov 02 22:59:06.672265 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:06.672105 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f674fe20-ea05-4246-a4ff-7cd1da321621-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-b986v\" (UID: \"f674fe20-ea05-4246-a4ff-7cd1da321621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b986v" Nov 02 22:59:06.672265 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:06.672204 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f674fe20-ea05-4246-a4ff-7cd1da321621-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-b986v\" (UID: \"f674fe20-ea05-4246-a4ff-7cd1da321621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b986v" Nov 02 22:59:06.674028 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:06.674011 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f674fe20-ea05-4246-a4ff-7cd1da321621-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-b986v\" (UID: \"f674fe20-ea05-4246-a4ff-7cd1da321621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b986v" Nov 02 22:59:06.674310 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:06.674293 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f674fe20-ea05-4246-a4ff-7cd1da321621-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-b986v\" (UID: \"f674fe20-ea05-4246-a4ff-7cd1da321621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b986v" Nov 02 22:59:06.732428 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:06.732356 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gzg2\" (UniqueName: \"kubernetes.io/projected/f674fe20-ea05-4246-a4ff-7cd1da321621-kube-api-access-5gzg2\") pod \"echo-simulator-kserve-64b6f8f96c-b986v\" (UID: \"f674fe20-ea05-4246-a4ff-7cd1da321621\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b986v" Nov 02 22:59:06.748294 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:06.748272 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-b986v" Nov 02 22:59:07.130606 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:07.130584 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-b986v"] Nov 02 22:59:07.132100 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:59:07.132072 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf674fe20_ea05_4246_a4ff_7cd1da321621.slice/crio-5374fec6a17b0833d6d9ce00db8bda0642e8734f0ec0426b2f9a932a1273c4df WatchSource:0}: Error finding container 5374fec6a17b0833d6d9ce00db8bda0642e8734f0ec0426b2f9a932a1273c4df: Status 404 returned error can't find the container with id 5374fec6a17b0833d6d9ce00db8bda0642e8734f0ec0426b2f9a932a1273c4df Nov 02 22:59:07.133617 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:07.133601 5727 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:59:07.599454 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:07.599419 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-b986v" event={"ID":"f674fe20-ea05-4246-a4ff-7cd1da321621","Type":"ContainerStarted","Data":"5374fec6a17b0833d6d9ce00db8bda0642e8734f0ec0426b2f9a932a1273c4df"} Nov 02 22:59:10.517510 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:10.517480 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4"] Nov 02 22:59:10.521649 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:10.521623 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4" Nov 02 22:59:10.611223 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:10.611192 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/359b19c7-a9c6-4476-be2d-9ab428699c1c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4\" (UID: \"359b19c7-a9c6-4476-be2d-9ab428699c1c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4" Nov 02 22:59:10.611387 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:10.611338 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/359b19c7-a9c6-4476-be2d-9ab428699c1c-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4\" (UID: \"359b19c7-a9c6-4476-be2d-9ab428699c1c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4" Nov 02 22:59:10.611436 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:10.611395 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/359b19c7-a9c6-4476-be2d-9ab428699c1c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4\" (UID: \"359b19c7-a9c6-4476-be2d-9ab428699c1c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4" Nov 02 22:59:10.611499 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:10.611447 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/359b19c7-a9c6-4476-be2d-9ab428699c1c-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4\" (UID: \"359b19c7-a9c6-4476-be2d-9ab428699c1c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4" Nov 02 22:59:10.611550 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:10.611530 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/359b19c7-a9c6-4476-be2d-9ab428699c1c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4\" (UID: \"359b19c7-a9c6-4476-be2d-9ab428699c1c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4" Nov 02 22:59:10.611636 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:10.611620 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mfkk\" (UniqueName: \"kubernetes.io/projected/359b19c7-a9c6-4476-be2d-9ab428699c1c-kube-api-access-6mfkk\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4\" (UID: \"359b19c7-a9c6-4476-be2d-9ab428699c1c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4" Nov 02 22:59:10.621803 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:10.621776 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-b986v" event={"ID":"f674fe20-ea05-4246-a4ff-7cd1da321621","Type":"ContainerStarted","Data":"7b0ef149fce944c8c18cd86fb187b83a79626ba7533ef9525b6d11c8fb6d5146"} Nov 02 22:59:10.622938 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:10.622918 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 02 22:59:10.625344 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:10.625324 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4"] Nov 02 22:59:10.713088 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:10.713061 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/359b19c7-a9c6-4476-be2d-9ab428699c1c-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4\" (UID: \"359b19c7-a9c6-4476-be2d-9ab428699c1c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4" Nov 02 22:59:10.713088 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:10.713092 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/359b19c7-a9c6-4476-be2d-9ab428699c1c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4\" (UID: \"359b19c7-a9c6-4476-be2d-9ab428699c1c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4" Nov 02 22:59:10.713395 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:10.713370 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/359b19c7-a9c6-4476-be2d-9ab428699c1c-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4\" (UID: \"359b19c7-a9c6-4476-be2d-9ab428699c1c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4" Nov 02 22:59:10.713460 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:10.713410 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/359b19c7-a9c6-4476-be2d-9ab428699c1c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4\" (UID: \"359b19c7-a9c6-4476-be2d-9ab428699c1c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4" Nov 02 22:59:10.713460 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:10.713442 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mfkk\" (UniqueName: \"kubernetes.io/projected/359b19c7-a9c6-4476-be2d-9ab428699c1c-kube-api-access-6mfkk\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4\" (UID: \"359b19c7-a9c6-4476-be2d-9ab428699c1c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4" Nov 02 22:59:10.713562 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:10.713544 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/359b19c7-a9c6-4476-be2d-9ab428699c1c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4\" (UID: \"359b19c7-a9c6-4476-be2d-9ab428699c1c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4" Nov 02 22:59:10.713626 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:10.713585 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/359b19c7-a9c6-4476-be2d-9ab428699c1c-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4\" (UID: \"359b19c7-a9c6-4476-be2d-9ab428699c1c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4" Nov 02 22:59:10.713745 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:10.713721 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/359b19c7-a9c6-4476-be2d-9ab428699c1c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4\" (UID: \"359b19c7-a9c6-4476-be2d-9ab428699c1c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4" Nov 02 22:59:10.714021 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:10.714001 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/359b19c7-a9c6-4476-be2d-9ab428699c1c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4\" (UID: \"359b19c7-a9c6-4476-be2d-9ab428699c1c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4" Nov 02 22:59:10.715770 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:10.715750 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/359b19c7-a9c6-4476-be2d-9ab428699c1c-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4\" (UID: \"359b19c7-a9c6-4476-be2d-9ab428699c1c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4" Nov 02 22:59:10.716174 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:10.716152 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/359b19c7-a9c6-4476-be2d-9ab428699c1c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4\" (UID: \"359b19c7-a9c6-4476-be2d-9ab428699c1c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4" Nov 02 22:59:10.748986 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:10.748964 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-b986v" Nov 02 22:59:10.754361 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:10.754320 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-b986v" podStartSLOduration=2.036981697 podStartE2EDuration="4.754307839s" totalImagesPullingTime="2.717326142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:59:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:59:10.752178901 +0000 UTC m=+636.990676587" watchObservedRunningTime="2026-11-02 22:59:10.754307839 +0000 UTC m=+636.992805543" Nov 02 22:59:10.773978 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:10.773928 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mfkk\" (UniqueName: \"kubernetes.io/projected/359b19c7-a9c6-4476-be2d-9ab428699c1c-kube-api-access-6mfkk\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4\" (UID: \"359b19c7-a9c6-4476-be2d-9ab428699c1c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4" Nov 02 22:59:10.834925 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:10.834894 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4" Nov 02 22:59:11.234757 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:11.234727 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4"] Nov 02 22:59:11.236953 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:59:11.236912 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod359b19c7_a9c6_4476_be2d_9ab428699c1c.slice/crio-577415a688b496eaad75c0eb2578943ad3adf8fd1cb69b353bbe51be3faad9e1 WatchSource:0}: Error finding container 577415a688b496eaad75c0eb2578943ad3adf8fd1cb69b353bbe51be3faad9e1: Status 404 returned error can't find the container with id 577415a688b496eaad75c0eb2578943ad3adf8fd1cb69b353bbe51be3faad9e1 Nov 02 22:59:11.629975 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:11.629937 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4" event={"ID":"359b19c7-a9c6-4476-be2d-9ab428699c1c","Type":"ContainerStarted","Data":"577415a688b496eaad75c0eb2578943ad3adf8fd1cb69b353bbe51be3faad9e1"} Nov 02 22:59:12.637929 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:12.637894 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4" event={"ID":"359b19c7-a9c6-4476-be2d-9ab428699c1c","Type":"ContainerStarted","Data":"0f2e39aba5ebad9d76071540e5fa177b0ba20bc86db05b2478f3edb17dafd196"} Nov 02 22:59:12.757121 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:12.757047 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4" podStartSLOduration=2.055835293 podStartE2EDuration="2.757029536s" totalImagesPullingTime="701.194243ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:59:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:59:12.755024724 +0000 UTC m=+638.993522386" watchObservedRunningTime="2026-11-02 22:59:12.757029536 +0000 UTC m=+638.995527199" Nov 02 22:59:12.835513 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:12.835471 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4" Nov 02 22:59:16.749266 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:16.749184 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-b986v" Nov 02 22:59:16.761829 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:16.761806 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-b986v" Nov 02 22:59:17.685406 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:17.685365 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-b986v" Nov 02 22:59:20.835725 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:20.835695 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4" Nov 02 22:59:20.848244 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:20.848223 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4" Nov 02 22:59:21.708348 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:21.708315 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4" Nov 02 22:59:23.425003 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:23.424970 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd"] Nov 02 22:59:23.428868 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:23.428849 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:23.533807 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:23.533776 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd"] Nov 02 22:59:23.539217 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:23.539190 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d966bed-7db8-4d7f-aad4-a9f00d168242-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-7dzqd\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:23.539352 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:23.539246 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-7dzqd\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:23.539352 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:23.539272 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-7dzqd\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:23.539454 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:23.539403 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d966bed-7db8-4d7f-aad4-a9f00d168242-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-7dzqd\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:23.539454 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:23.539427 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-7dzqd\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:23.539538 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:23.539498 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-7dzqd\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:23.539577 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:23.539548 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-7dzqd\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:23.539622 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:23.539599 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcqkr\" (UniqueName: \"kubernetes.io/projected/0d966bed-7db8-4d7f-aad4-a9f00d168242-kube-api-access-zcqkr\") pod \"maas-default-gateway-openshift-default-694db55c87-7dzqd\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:23.539672 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:23.539619 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d966bed-7db8-4d7f-aad4-a9f00d168242-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-7dzqd\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:23.640968 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:23.640934 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d966bed-7db8-4d7f-aad4-a9f00d168242-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-7dzqd\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:23.641183 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:23.640975 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d966bed-7db8-4d7f-aad4-a9f00d168242-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-7dzqd\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:23.641183 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:23.641018 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-7dzqd\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:23.641183 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:23.641038 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-7dzqd\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:23.641183 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:23.641056 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d966bed-7db8-4d7f-aad4-a9f00d168242-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-7dzqd\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:23.641183 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:23.641074 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-7dzqd\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:23.641183 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:23.641158 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-7dzqd\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:23.641543 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:23.641201 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-7dzqd\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:23.641543 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:23.641260 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcqkr\" (UniqueName: \"kubernetes.io/projected/0d966bed-7db8-4d7f-aad4-a9f00d168242-kube-api-access-zcqkr\") pod \"maas-default-gateway-openshift-default-694db55c87-7dzqd\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:23.641543 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:23.641488 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-7dzqd\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:23.641722 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:23.641589 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-7dzqd\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:23.641722 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:23.641621 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-7dzqd\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:23.641831 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:23.641801 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-7dzqd\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:23.641990 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:23.641967 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d966bed-7db8-4d7f-aad4-a9f00d168242-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-7dzqd\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:23.643637 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:23.643618 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d966bed-7db8-4d7f-aad4-a9f00d168242-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-7dzqd\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:23.643750 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:23.643707 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-7dzqd\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:23.701518 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:23.701451 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d966bed-7db8-4d7f-aad4-a9f00d168242-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-7dzqd\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:23.701876 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:23.701857 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcqkr\" (UniqueName: \"kubernetes.io/projected/0d966bed-7db8-4d7f-aad4-a9f00d168242-kube-api-access-zcqkr\") pod \"maas-default-gateway-openshift-default-694db55c87-7dzqd\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:23.741545 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:23.741522 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:24.167164 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:24.167139 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd"] Nov 02 22:59:24.169446 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:59:24.169420 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d966bed_7db8_4d7f_aad4_a9f00d168242.slice/crio-4dca1d914ea782047cac222b9a19ff5b1338e322a5bd0fc27574359e1bc4f77d WatchSource:0}: Error finding container 4dca1d914ea782047cac222b9a19ff5b1338e322a5bd0fc27574359e1bc4f77d: Status 404 returned error can't find the container with id 4dca1d914ea782047cac222b9a19ff5b1338e322a5bd0fc27574359e1bc4f77d Nov 02 22:59:24.171591 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:24.171553 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 22:59:24.171710 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:24.171624 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 22:59:24.171710 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:24.171656 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 22:59:24.716678 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:24.716643 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" event={"ID":"0d966bed-7db8-4d7f-aad4-a9f00d168242","Type":"ContainerStarted","Data":"7028878b2e5bd4c61623408018f7f1388b31c5e3b2b743beacb534ca46590ec8"} Nov 02 22:59:24.717092 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:24.716704 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" event={"ID":"0d966bed-7db8-4d7f-aad4-a9f00d168242","Type":"ContainerStarted","Data":"4dca1d914ea782047cac222b9a19ff5b1338e322a5bd0fc27574359e1bc4f77d"} Nov 02 22:59:24.741979 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:24.741953 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:24.839195 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:24.839113 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" podStartSLOduration=1.839096873 podStartE2EDuration="1.839096873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:59:24.837742694 +0000 UTC m=+651.076240357" watchObservedRunningTime="2026-11-02 22:59:24.839096873 +0000 UTC m=+651.077594535" Nov 02 22:59:25.743732 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:25.743694 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-7dzqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 02 22:59:25.744112 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:25.743751 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" podUID="0d966bed-7db8-4d7f-aad4-a9f00d168242" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 02 22:59:26.742680 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:26.742647 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-7dzqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 02 22:59:26.742857 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:26.742703 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" podUID="0d966bed-7db8-4d7f-aad4-a9f00d168242" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 02 22:59:27.742267 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:27.742234 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-7dzqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 02 22:59:27.742655 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:27.742283 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" podUID="0d966bed-7db8-4d7f-aad4-a9f00d168242" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 02 22:59:27.845113 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:59:27.845076 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 22:59:28.742472 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:28.742439 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-7dzqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 02 22:59:28.742875 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:28.742490 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" podUID="0d966bed-7db8-4d7f-aad4-a9f00d168242" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 02 22:59:29.742096 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:29.742061 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-7dzqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 02 22:59:29.742304 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:29.742116 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" podUID="0d966bed-7db8-4d7f-aad4-a9f00d168242" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 02 22:59:30.741787 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:30.741754 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-7dzqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 02 22:59:30.742199 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:30.741802 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" podUID="0d966bed-7db8-4d7f-aad4-a9f00d168242" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 02 22:59:32.370826 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:32.370791 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:32.371278 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:32.371225 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:32.371928 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:32.371911 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 22:59:40.844348 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:40.844315 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f5d89f7c-zkm62"] Nov 02 22:59:40.853322 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:40.853298 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f5d89f7c-zkm62" Nov 02 22:59:40.903597 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:40.903573 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f5d89f7c-zkm62"] Nov 02 22:59:40.926065 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:40.926041 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cf7f72a3-5ca3-4f5e-a8e2-c88f1485806a-tls-cert\") pod \"authorino-57f5d89f7c-zkm62\" (UID: \"cf7f72a3-5ca3-4f5e-a8e2-c88f1485806a\") " pod="kuadrant-system/authorino-57f5d89f7c-zkm62" Nov 02 22:59:40.926197 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:40.926113 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg4zp\" (UniqueName: \"kubernetes.io/projected/cf7f72a3-5ca3-4f5e-a8e2-c88f1485806a-kube-api-access-jg4zp\") pod \"authorino-57f5d89f7c-zkm62\" (UID: \"cf7f72a3-5ca3-4f5e-a8e2-c88f1485806a\") " pod="kuadrant-system/authorino-57f5d89f7c-zkm62" Nov 02 22:59:41.026804 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:41.026776 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg4zp\" (UniqueName: \"kubernetes.io/projected/cf7f72a3-5ca3-4f5e-a8e2-c88f1485806a-kube-api-access-jg4zp\") pod \"authorino-57f5d89f7c-zkm62\" (UID: \"cf7f72a3-5ca3-4f5e-a8e2-c88f1485806a\") " pod="kuadrant-system/authorino-57f5d89f7c-zkm62" Nov 02 22:59:41.026942 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:41.026855 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cf7f72a3-5ca3-4f5e-a8e2-c88f1485806a-tls-cert\") pod \"authorino-57f5d89f7c-zkm62\" (UID: \"cf7f72a3-5ca3-4f5e-a8e2-c88f1485806a\") " pod="kuadrant-system/authorino-57f5d89f7c-zkm62" Nov 02 22:59:41.029397 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:41.029368 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cf7f72a3-5ca3-4f5e-a8e2-c88f1485806a-tls-cert\") pod \"authorino-57f5d89f7c-zkm62\" (UID: \"cf7f72a3-5ca3-4f5e-a8e2-c88f1485806a\") " pod="kuadrant-system/authorino-57f5d89f7c-zkm62" Nov 02 22:59:41.087870 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:41.087845 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg4zp\" (UniqueName: \"kubernetes.io/projected/cf7f72a3-5ca3-4f5e-a8e2-c88f1485806a-kube-api-access-jg4zp\") pod \"authorino-57f5d89f7c-zkm62\" (UID: \"cf7f72a3-5ca3-4f5e-a8e2-c88f1485806a\") " pod="kuadrant-system/authorino-57f5d89f7c-zkm62" Nov 02 22:59:41.166361 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:41.166298 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f5d89f7c-zkm62" Nov 02 22:59:41.541484 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:41.541460 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f5d89f7c-zkm62"] Nov 02 22:59:41.544292 ip-10-0-135-176 kubenswrapper[5727]: W1102 22:59:41.544265 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf7f72a3_5ca3_4f5e_a8e2_c88f1485806a.slice/crio-6e67961973c8b7a8bb69c8942a92f7064658eccdf3f6cb5f8cd0082579629baa WatchSource:0}: Error finding container 6e67961973c8b7a8bb69c8942a92f7064658eccdf3f6cb5f8cd0082579629baa: Status 404 returned error can't find the container with id 6e67961973c8b7a8bb69c8942a92f7064658eccdf3f6cb5f8cd0082579629baa Nov 02 22:59:41.816029 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:41.815995 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f5d89f7c-zkm62" event={"ID":"cf7f72a3-5ca3-4f5e-a8e2-c88f1485806a","Type":"ContainerStarted","Data":"6e67961973c8b7a8bb69c8942a92f7064658eccdf3f6cb5f8cd0082579629baa"} Nov 02 22:59:43.828797 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:43.828758 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f5d89f7c-zkm62" event={"ID":"cf7f72a3-5ca3-4f5e-a8e2-c88f1485806a","Type":"ContainerStarted","Data":"7e0f659ee53e893fb46b3479ce0d1cd9443d051706cb6fa38da550dbb7924239"} Nov 02 22:59:43.944781 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:43.944727 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f5d89f7c-zkm62" podStartSLOduration=2.425288588 podStartE2EDuration="3.944713652s" totalImagesPullingTime="1.519425064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:59:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:59:43.942366586 +0000 UTC m=+670.180864269" watchObservedRunningTime="2026-11-02 22:59:43.944713652 +0000 UTC m=+670.183211313" Nov 02 22:59:43.979277 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:43.979245 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-78f5fdb9b6-bmxbs"] Nov 02 22:59:43.979472 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:43.979437 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-78f5fdb9b6-bmxbs" podUID="9cb33a7f-dd45-41ad-9b78-45bf297a29d8" containerName="authorino" containerID="cri-o://fcb9b338f67b5e3c00e7b4038ecabb8c8eb8f7e6a755e1712d89d98fb43e95d3" gracePeriod=30 Nov 02 22:59:44.254108 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:44.254082 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78f5fdb9b6-bmxbs" Nov 02 22:59:44.358784 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:44.358706 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cb33a7f-dd45-41ad-9b78-45bf297a29d8-tls-cert\" (UniqueName: \"kubernetes.io/secret/9cb33a7f-dd45-41ad-9b78-45bf297a29d8-tls-cert\") pod \"9cb33a7f-dd45-41ad-9b78-45bf297a29d8\" (UID: \"9cb33a7f-dd45-41ad-9b78-45bf297a29d8\") " Nov 02 22:59:44.358928 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:44.358791 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cb33a7f-dd45-41ad-9b78-45bf297a29d8-kube-api-access-vq7nb\" (UniqueName: \"kubernetes.io/projected/9cb33a7f-dd45-41ad-9b78-45bf297a29d8-kube-api-access-vq7nb\") pod \"9cb33a7f-dd45-41ad-9b78-45bf297a29d8\" (UID: \"9cb33a7f-dd45-41ad-9b78-45bf297a29d8\") " Nov 02 22:59:44.361176 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:44.361118 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb33a7f-dd45-41ad-9b78-45bf297a29d8-kube-api-access-vq7nb" pod "9cb33a7f-dd45-41ad-9b78-45bf297a29d8" (UID: "9cb33a7f-dd45-41ad-9b78-45bf297a29d8"). InnerVolumeSpecName "kube-api-access-vq7nb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:59:44.394466 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:44.394412 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cb33a7f-dd45-41ad-9b78-45bf297a29d8-tls-cert" pod "9cb33a7f-dd45-41ad-9b78-45bf297a29d8" (UID: "9cb33a7f-dd45-41ad-9b78-45bf297a29d8"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:59:44.460024 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:44.459990 5727 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9cb33a7f-dd45-41ad-9b78-45bf297a29d8-tls-cert\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:59:44.460024 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:44.460020 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq7nb\" (UniqueName: \"kubernetes.io/projected/9cb33a7f-dd45-41ad-9b78-45bf297a29d8-kube-api-access-vq7nb\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:59:44.836621 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:44.836585 5727 generic.go:356] "Generic (PLEG): container finished" podID="9cb33a7f-dd45-41ad-9b78-45bf297a29d8" containerID="fcb9b338f67b5e3c00e7b4038ecabb8c8eb8f7e6a755e1712d89d98fb43e95d3" exitCode=0 Nov 02 22:59:44.837113 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:44.836632 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78f5fdb9b6-bmxbs" Nov 02 22:59:44.837113 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:44.836667 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78f5fdb9b6-bmxbs" event={"ID":"9cb33a7f-dd45-41ad-9b78-45bf297a29d8","Type":"ContainerDied","Data":"fcb9b338f67b5e3c00e7b4038ecabb8c8eb8f7e6a755e1712d89d98fb43e95d3"} Nov 02 22:59:44.837113 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:44.836709 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78f5fdb9b6-bmxbs" event={"ID":"9cb33a7f-dd45-41ad-9b78-45bf297a29d8","Type":"ContainerDied","Data":"07cfc61808e97707611e8bfcdc1c9a1ca05719831d84c1d87e6e0b974020bef4"} Nov 02 22:59:44.837113 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:44.836725 5727 scope.go:122] "RemoveContainer" containerID="fcb9b338f67b5e3c00e7b4038ecabb8c8eb8f7e6a755e1712d89d98fb43e95d3" Nov 02 22:59:44.852817 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:44.852790 5727 scope.go:122] "RemoveContainer" containerID="fcb9b338f67b5e3c00e7b4038ecabb8c8eb8f7e6a755e1712d89d98fb43e95d3" Nov 02 22:59:44.853162 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:59:44.853115 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcb9b338f67b5e3c00e7b4038ecabb8c8eb8f7e6a755e1712d89d98fb43e95d3\": container with ID starting with fcb9b338f67b5e3c00e7b4038ecabb8c8eb8f7e6a755e1712d89d98fb43e95d3 not found: ID does not exist" containerID="fcb9b338f67b5e3c00e7b4038ecabb8c8eb8f7e6a755e1712d89d98fb43e95d3" Nov 02 22:59:44.853231 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:44.853172 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcb9b338f67b5e3c00e7b4038ecabb8c8eb8f7e6a755e1712d89d98fb43e95d3"} err="failed to get container status \"fcb9b338f67b5e3c00e7b4038ecabb8c8eb8f7e6a755e1712d89d98fb43e95d3\": rpc error: code = NotFound desc = could not find container \"fcb9b338f67b5e3c00e7b4038ecabb8c8eb8f7e6a755e1712d89d98fb43e95d3\": container with ID starting with fcb9b338f67b5e3c00e7b4038ecabb8c8eb8f7e6a755e1712d89d98fb43e95d3 not found: ID does not exist" Nov 02 22:59:45.002823 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:45.002785 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-78f5fdb9b6-bmxbs"] Nov 02 22:59:45.008217 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:45.008183 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-78f5fdb9b6-bmxbs"] Nov 02 22:59:45.883860 ip-10-0-135-176 kubenswrapper[5727]: I1102 22:59:45.883824 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cb33a7f-dd45-41ad-9b78-45bf297a29d8" path="/var/lib/kubelet/pods/9cb33a7f-dd45-41ad-9b78-45bf297a29d8/volumes" Nov 02 22:59:57.845013 ip-10-0-135-176 kubenswrapper[5727]: E1102 22:59:57.844977 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:00:00.178598 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:00.178560 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894340-l5srw"] Nov 02 23:00:00.179023 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:00.179010 5727 cpu_manager.go:414] "containerMap: removing container" podUID="9cb33a7f-dd45-41ad-9b78-45bf297a29d8" containerName="authorino" Nov 02 23:00:00.179067 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:00.179023 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cb33a7f-dd45-41ad-9b78-45bf297a29d8" containerName="authorino" Nov 02 23:00:00.179118 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:00.179109 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cb33a7f-dd45-41ad-9b78-45bf297a29d8" containerName="authorino" Nov 02 23:00:00.182441 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:00.182426 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-l5srw" Nov 02 23:00:00.239791 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:00.239771 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894340-l5srw"] Nov 02 23:00:00.286568 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:00.286543 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:00:00.286714 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:00.286603 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:00:00.286714 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:00.286697 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:00:00.321597 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:00.321574 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c75bf587-6f92-43c4-b7da-f1cd5afad678-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894340-l5srw\" (UID: \"c75bf587-6f92-43c4-b7da-f1cd5afad678\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-l5srw" Nov 02 23:00:00.321699 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:00.321630 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw62m\" (UniqueName: \"kubernetes.io/projected/c75bf587-6f92-43c4-b7da-f1cd5afad678-kube-api-access-bw62m\") pod \"maas-api-key-cleanup-29894340-l5srw\" (UID: \"c75bf587-6f92-43c4-b7da-f1cd5afad678\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-l5srw" Nov 02 23:00:00.422093 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:00.422064 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c75bf587-6f92-43c4-b7da-f1cd5afad678-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894340-l5srw\" (UID: \"c75bf587-6f92-43c4-b7da-f1cd5afad678\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-l5srw" Nov 02 23:00:00.422219 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:00.422163 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw62m\" (UniqueName: \"kubernetes.io/projected/c75bf587-6f92-43c4-b7da-f1cd5afad678-kube-api-access-bw62m\") pod \"maas-api-key-cleanup-29894340-l5srw\" (UID: \"c75bf587-6f92-43c4-b7da-f1cd5afad678\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-l5srw" Nov 02 23:00:00.422685 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:00.422669 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c75bf587-6f92-43c4-b7da-f1cd5afad678-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894340-l5srw\" (UID: \"c75bf587-6f92-43c4-b7da-f1cd5afad678\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-l5srw" Nov 02 23:00:00.483427 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:00.483363 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw62m\" (UniqueName: \"kubernetes.io/projected/c75bf587-6f92-43c4-b7da-f1cd5afad678-kube-api-access-bw62m\") pod \"maas-api-key-cleanup-29894340-l5srw\" (UID: \"c75bf587-6f92-43c4-b7da-f1cd5afad678\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-l5srw" Nov 02 23:00:00.495033 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:00.495010 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-l5srw" Nov 02 23:00:00.870072 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:00.870050 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894340-l5srw"] Nov 02 23:00:00.871810 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:00:00.871783 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc75bf587_6f92_43c4_b7da_f1cd5afad678.slice/crio-61131360e5a5f1d2037750c81308950648bd74a4c03c1640e0abe89bbb5a2a50 WatchSource:0}: Error finding container 61131360e5a5f1d2037750c81308950648bd74a4c03c1640e0abe89bbb5a2a50: Status 404 returned error can't find the container with id 61131360e5a5f1d2037750c81308950648bd74a4c03c1640e0abe89bbb5a2a50 Nov 02 23:00:00.941606 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:00.941579 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-l5srw" event={"ID":"c75bf587-6f92-43c4-b7da-f1cd5afad678","Type":"ContainerStarted","Data":"61131360e5a5f1d2037750c81308950648bd74a4c03c1640e0abe89bbb5a2a50"} Nov 02 23:00:05.975375 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:05.975341 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-l5srw" event={"ID":"c75bf587-6f92-43c4-b7da-f1cd5afad678","Type":"ContainerStarted","Data":"f02896597885c734cb99f36a58c0396170b1daf9814f70b8aac3219e4d8875af"} Nov 02 23:00:06.093353 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:06.093299 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-l5srw" podStartSLOduration=1.451732179 podStartE2EDuration="6.093285081s" totalImagesPullingTime="4.641552902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:00:06.090200532 +0000 UTC m=+692.328698214" watchObservedRunningTime="2026-11-02 23:00:06.093285081 +0000 UTC m=+692.331782802" Nov 02 23:00:06.984198 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:06.984155 5727 generic.go:356] "Generic (PLEG): container finished" podID="c75bf587-6f92-43c4-b7da-f1cd5afad678" containerID="f02896597885c734cb99f36a58c0396170b1daf9814f70b8aac3219e4d8875af" exitCode=60 Nov 02 23:00:06.984577 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:06.984210 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-l5srw" event={"ID":"c75bf587-6f92-43c4-b7da-f1cd5afad678","Type":"ContainerDied","Data":"f02896597885c734cb99f36a58c0396170b1daf9814f70b8aac3219e4d8875af"} Nov 02 23:00:06.984577 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:06.984439 5727 scope.go:122] "RemoveContainer" containerID="f02896597885c734cb99f36a58c0396170b1daf9814f70b8aac3219e4d8875af" Nov 02 23:00:07.992303 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:07.992272 5727 generic.go:356] "Generic (PLEG): container finished" podID="c75bf587-6f92-43c4-b7da-f1cd5afad678" containerID="b962573b5cfd9a8e4978436bf79efec9f2a08ad4f9141d3858dbd76c363d31d4" exitCode=60 Nov 02 23:00:07.992669 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:07.992351 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-l5srw" event={"ID":"c75bf587-6f92-43c4-b7da-f1cd5afad678","Type":"ContainerDied","Data":"b962573b5cfd9a8e4978436bf79efec9f2a08ad4f9141d3858dbd76c363d31d4"} Nov 02 23:00:07.992669 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:07.992397 5727 scope.go:122] "RemoveContainer" containerID="f02896597885c734cb99f36a58c0396170b1daf9814f70b8aac3219e4d8875af" Nov 02 23:00:07.992669 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:07.992603 5727 scope.go:122] "RemoveContainer" containerID="b962573b5cfd9a8e4978436bf79efec9f2a08ad4f9141d3858dbd76c363d31d4" Nov 02 23:00:07.992867 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:00:07.992845 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894340-l5srw_odh-ai-gateway-infra(c75bf587-6f92-43c4-b7da-f1cd5afad678)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-l5srw" podUID="c75bf587-6f92-43c4-b7da-f1cd5afad678" Nov 02 23:00:08.999900 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:08.999869 5727 scope.go:122] "RemoveContainer" containerID="b962573b5cfd9a8e4978436bf79efec9f2a08ad4f9141d3858dbd76c363d31d4" Nov 02 23:00:09.000311 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:00:09.000091 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894340-l5srw_odh-ai-gateway-infra(c75bf587-6f92-43c4-b7da-f1cd5afad678)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-l5srw" podUID="c75bf587-6f92-43c4-b7da-f1cd5afad678" Nov 02 23:00:20.833366 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:20.833330 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fed9e8ee-644dc48579-kvq2w"] Nov 02 23:00:20.837123 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:20.837102 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed9e8ee-644dc48579-kvq2w" Nov 02 23:00:20.875508 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:20.875486 5727 scope.go:122] "RemoveContainer" containerID="b962573b5cfd9a8e4978436bf79efec9f2a08ad4f9141d3858dbd76c363d31d4" Nov 02 23:00:20.905487 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:20.905419 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fed9e8ee-644dc48579-kvq2w"] Nov 02 23:00:20.926369 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:20.917296 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7ccf\" (UniqueName: \"kubernetes.io/projected/443ff9dc-8fc9-447a-b0e4-1606cd671b61-kube-api-access-b7ccf\") pod \"payload-pre-processing-e2e-migrate-fed9e8ee-644dc48579-kvq2w\" (UID: \"443ff9dc-8fc9-447a-b0e4-1606cd671b61\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed9e8ee-644dc48579-kvq2w" Nov 02 23:00:20.926369 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:20.917356 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/443ff9dc-8fc9-447a-b0e4-1606cd671b61-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fed9e8ee-644dc48579-kvq2w\" (UID: \"443ff9dc-8fc9-447a-b0e4-1606cd671b61\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed9e8ee-644dc48579-kvq2w" Nov 02 23:00:20.949169 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:20.949123 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-fed9e8ee\"" Nov 02 23:00:20.949325 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:20.949177 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-fed9e8ee-dockercfg-2mrdq\"" Nov 02 23:00:21.001463 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:21.001436 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fed9e8ee-787876d555-j2dhc"] Nov 02 23:00:21.006311 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:21.006294 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fed9e8ee-787876d555-j2dhc" Nov 02 23:00:21.018512 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:21.018487 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr2fh\" (UniqueName: \"kubernetes.io/projected/b32b75be-1f34-4ca0-8494-dd6bf7f31524-kube-api-access-pr2fh\") pod \"payload-processing-e2e-migrate-fed9e8ee-787876d555-j2dhc\" (UID: \"b32b75be-1f34-4ca0-8494-dd6bf7f31524\") " pod="openshift-ingress/payload-processing-e2e-migrate-fed9e8ee-787876d555-j2dhc" Nov 02 23:00:21.018613 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:21.018526 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b32b75be-1f34-4ca0-8494-dd6bf7f31524-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fed9e8ee-787876d555-j2dhc\" (UID: \"b32b75be-1f34-4ca0-8494-dd6bf7f31524\") " pod="openshift-ingress/payload-processing-e2e-migrate-fed9e8ee-787876d555-j2dhc" Nov 02 23:00:21.018696 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:21.018676 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7ccf\" (UniqueName: \"kubernetes.io/projected/443ff9dc-8fc9-447a-b0e4-1606cd671b61-kube-api-access-b7ccf\") pod \"payload-pre-processing-e2e-migrate-fed9e8ee-644dc48579-kvq2w\" (UID: \"443ff9dc-8fc9-447a-b0e4-1606cd671b61\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed9e8ee-644dc48579-kvq2w" Nov 02 23:00:21.018783 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:21.018762 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/443ff9dc-8fc9-447a-b0e4-1606cd671b61-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fed9e8ee-644dc48579-kvq2w\" (UID: \"443ff9dc-8fc9-447a-b0e4-1606cd671b61\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed9e8ee-644dc48579-kvq2w" Nov 02 23:00:21.019449 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:21.019429 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/443ff9dc-8fc9-447a-b0e4-1606cd671b61-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fed9e8ee-644dc48579-kvq2w\" (UID: \"443ff9dc-8fc9-447a-b0e4-1606cd671b61\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed9e8ee-644dc48579-kvq2w" Nov 02 23:00:21.057805 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:21.057778 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fed9e8ee-787876d555-j2dhc"] Nov 02 23:00:21.080356 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:21.080333 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7ccf\" (UniqueName: \"kubernetes.io/projected/443ff9dc-8fc9-447a-b0e4-1606cd671b61-kube-api-access-b7ccf\") pod \"payload-pre-processing-e2e-migrate-fed9e8ee-644dc48579-kvq2w\" (UID: \"443ff9dc-8fc9-447a-b0e4-1606cd671b61\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed9e8ee-644dc48579-kvq2w" Nov 02 23:00:21.119607 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:21.119347 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr2fh\" (UniqueName: \"kubernetes.io/projected/b32b75be-1f34-4ca0-8494-dd6bf7f31524-kube-api-access-pr2fh\") pod \"payload-processing-e2e-migrate-fed9e8ee-787876d555-j2dhc\" (UID: \"b32b75be-1f34-4ca0-8494-dd6bf7f31524\") " pod="openshift-ingress/payload-processing-e2e-migrate-fed9e8ee-787876d555-j2dhc" Nov 02 23:00:21.119607 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:21.119398 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b32b75be-1f34-4ca0-8494-dd6bf7f31524-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fed9e8ee-787876d555-j2dhc\" (UID: \"b32b75be-1f34-4ca0-8494-dd6bf7f31524\") " pod="openshift-ingress/payload-processing-e2e-migrate-fed9e8ee-787876d555-j2dhc" Nov 02 23:00:21.120369 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:21.120346 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b32b75be-1f34-4ca0-8494-dd6bf7f31524-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fed9e8ee-787876d555-j2dhc\" (UID: \"b32b75be-1f34-4ca0-8494-dd6bf7f31524\") " pod="openshift-ingress/payload-processing-e2e-migrate-fed9e8ee-787876d555-j2dhc" Nov 02 23:00:21.150864 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:21.150842 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed9e8ee-644dc48579-kvq2w" Nov 02 23:00:21.181063 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:21.181043 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr2fh\" (UniqueName: \"kubernetes.io/projected/b32b75be-1f34-4ca0-8494-dd6bf7f31524-kube-api-access-pr2fh\") pod \"payload-processing-e2e-migrate-fed9e8ee-787876d555-j2dhc\" (UID: \"b32b75be-1f34-4ca0-8494-dd6bf7f31524\") " pod="openshift-ingress/payload-processing-e2e-migrate-fed9e8ee-787876d555-j2dhc" Nov 02 23:00:21.319335 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:21.319301 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fed9e8ee-787876d555-j2dhc" Nov 02 23:00:21.525684 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:21.525657 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fed9e8ee-644dc48579-kvq2w"] Nov 02 23:00:21.528387 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:00:21.528361 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod443ff9dc_8fc9_447a_b0e4_1606cd671b61.slice/crio-77fc931cecafd6ebd12c6ff62d194a7eeede36b0d92bc02543bd74334a3f7a23 WatchSource:0}: Error finding container 77fc931cecafd6ebd12c6ff62d194a7eeede36b0d92bc02543bd74334a3f7a23: Status 404 returned error can't find the container with id 77fc931cecafd6ebd12c6ff62d194a7eeede36b0d92bc02543bd74334a3f7a23 Nov 02 23:00:21.694400 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:21.694335 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fed9e8ee-787876d555-j2dhc"] Nov 02 23:00:21.720941 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:00:21.720915 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb32b75be_1f34_4ca0_8494_dd6bf7f31524.slice/crio-8950c35029ab561374e77641f70f2753582857ad2d8fa7318a553756a0e65622 WatchSource:0}: Error finding container 8950c35029ab561374e77641f70f2753582857ad2d8fa7318a553756a0e65622: Status 404 returned error can't find the container with id 8950c35029ab561374e77641f70f2753582857ad2d8fa7318a553756a0e65622 Nov 02 23:00:22.080560 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:22.080529 5727 generic.go:356] "Generic (PLEG): container finished" podID="c75bf587-6f92-43c4-b7da-f1cd5afad678" containerID="a6da73c0bc4134866691d0552e1fbb0682ce42ce32c8e9e8c0d445d813669acf" exitCode=60 Nov 02 23:00:22.080959 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:22.080601 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-l5srw" event={"ID":"c75bf587-6f92-43c4-b7da-f1cd5afad678","Type":"ContainerDied","Data":"a6da73c0bc4134866691d0552e1fbb0682ce42ce32c8e9e8c0d445d813669acf"} Nov 02 23:00:22.080959 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:22.080668 5727 scope.go:122] "RemoveContainer" containerID="b962573b5cfd9a8e4978436bf79efec9f2a08ad4f9141d3858dbd76c363d31d4" Nov 02 23:00:22.081087 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:22.080975 5727 scope.go:122] "RemoveContainer" containerID="a6da73c0bc4134866691d0552e1fbb0682ce42ce32c8e9e8c0d445d813669acf" Nov 02 23:00:22.081358 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:00:22.081199 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894340-l5srw_odh-ai-gateway-infra(c75bf587-6f92-43c4-b7da-f1cd5afad678)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-l5srw" podUID="c75bf587-6f92-43c4-b7da-f1cd5afad678" Nov 02 23:00:22.082516 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:22.082497 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fed9e8ee-787876d555-j2dhc" event={"ID":"b32b75be-1f34-4ca0-8494-dd6bf7f31524","Type":"ContainerStarted","Data":"5ecaa6b9b649f25d33156e6ed21c8bf36bf579a682dbd4292c4ec2af165acf8c"} Nov 02 23:00:22.082597 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:22.082542 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-fed9e8ee-787876d555-j2dhc" Nov 02 23:00:22.082597 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:22.082565 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fed9e8ee-787876d555-j2dhc" event={"ID":"b32b75be-1f34-4ca0-8494-dd6bf7f31524","Type":"ContainerStarted","Data":"8950c35029ab561374e77641f70f2753582857ad2d8fa7318a553756a0e65622"} Nov 02 23:00:22.083921 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:22.083903 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed9e8ee-644dc48579-kvq2w" event={"ID":"443ff9dc-8fc9-447a-b0e4-1606cd671b61","Type":"ContainerStarted","Data":"b3b2953d92b8da84617660f6476e3e7865af491654d8a24520f741b79b797e68"} Nov 02 23:00:22.083990 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:22.083940 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed9e8ee-644dc48579-kvq2w" event={"ID":"443ff9dc-8fc9-447a-b0e4-1606cd671b61","Type":"ContainerStarted","Data":"77fc931cecafd6ebd12c6ff62d194a7eeede36b0d92bc02543bd74334a3f7a23"} Nov 02 23:00:22.084048 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:22.084010 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed9e8ee-644dc48579-kvq2w" Nov 02 23:00:22.437583 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:22.437482 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-fed9e8ee-787876d555-j2dhc" podStartSLOduration=2.437466097 podStartE2EDuration="2.437466097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:00:22.32163909 +0000 UTC m=+708.560136751" watchObservedRunningTime="2026-11-02 23:00:22.437466097 +0000 UTC m=+708.675963758" Nov 02 23:00:22.437880 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:22.437856 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed9e8ee-644dc48579-kvq2w" podStartSLOduration=2.437851379 podStartE2EDuration="2.437851379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:00:22.435519715 +0000 UTC m=+708.674017398" watchObservedRunningTime="2026-11-02 23:00:22.437851379 +0000 UTC m=+708.676349041" Nov 02 23:00:22.677940 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:22.677911 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn"] Nov 02 23:00:22.683703 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:22.683681 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn" Nov 02 23:00:22.737231 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:22.737205 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-7d7bf49f-serving-cert\"" Nov 02 23:00:22.737369 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:22.737313 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wm4gz\"" Nov 02 23:00:22.737369 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:22.737355 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c-metrics-certs\") pod \"maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn\" (UID: \"2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn" Nov 02 23:00:22.737522 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:22.737477 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c-maas-api-tls\") pod \"maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn\" (UID: \"2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn" Nov 02 23:00:22.737581 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:22.737524 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 23:00:22.737581 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:22.737555 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mwv5\" (UniqueName: \"kubernetes.io/projected/2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c-kube-api-access-8mwv5\") pod \"maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn\" (UID: \"2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn" Nov 02 23:00:22.737680 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:22.737607 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c-openshift-service-ca\") pod \"maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn\" (UID: \"2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn" Nov 02 23:00:22.740700 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:22.740679 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn"] Nov 02 23:00:22.838255 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:22.838225 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c-metrics-certs\") pod \"maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn\" (UID: \"2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn" Nov 02 23:00:22.838426 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:22.838266 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c-maas-api-tls\") pod \"maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn\" (UID: \"2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn" Nov 02 23:00:22.838426 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:22.838387 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mwv5\" (UniqueName: \"kubernetes.io/projected/2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c-kube-api-access-8mwv5\") pod \"maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn\" (UID: \"2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn" Nov 02 23:00:22.838426 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:22.838423 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c-openshift-service-ca\") pod \"maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn\" (UID: \"2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn" Nov 02 23:00:22.838584 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:00:22.838526 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-7d7bf49f-serving-cert: secret "maas-api-e2e-ait-7d7bf49f-serving-cert" not found Nov 02 23:00:22.838643 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:00:22.838602 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c-maas-api-tls podName:2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c nodeName:}" failed. No retries permitted until 2026-11-02 23:00:23.338581353 +0000 UTC m=+709.577079009 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c-maas-api-tls") pod "maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn" (UID: "2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c") : secret "maas-api-e2e-ait-7d7bf49f-serving-cert" not found Nov 02 23:00:22.839064 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:22.839045 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c-openshift-service-ca\") pod \"maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn\" (UID: \"2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn" Nov 02 23:00:22.840912 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:22.840892 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c-metrics-certs\") pod \"maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn\" (UID: \"2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn" Nov 02 23:00:22.901692 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:22.901663 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mwv5\" (UniqueName: \"kubernetes.io/projected/2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c-kube-api-access-8mwv5\") pod \"maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn\" (UID: \"2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn" Nov 02 23:00:23.212338 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:23.212293 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894340-l5srw"] Nov 02 23:00:23.344261 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:23.344237 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c-maas-api-tls\") pod \"maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn\" (UID: \"2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn" Nov 02 23:00:23.346658 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:23.346637 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c-maas-api-tls\") pod \"maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn\" (UID: \"2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn" Nov 02 23:00:23.360858 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:23.360837 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-l5srw" Nov 02 23:00:23.444867 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:23.444837 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c75bf587-6f92-43c4-b7da-f1cd5afad678-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c75bf587-6f92-43c4-b7da-f1cd5afad678-maas-api-service-ca\") pod \"c75bf587-6f92-43c4-b7da-f1cd5afad678\" (UID: \"c75bf587-6f92-43c4-b7da-f1cd5afad678\") " Nov 02 23:00:23.444976 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:23.444925 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c75bf587-6f92-43c4-b7da-f1cd5afad678-kube-api-access-bw62m\" (UniqueName: \"kubernetes.io/projected/c75bf587-6f92-43c4-b7da-f1cd5afad678-kube-api-access-bw62m\") pod \"c75bf587-6f92-43c4-b7da-f1cd5afad678\" (UID: \"c75bf587-6f92-43c4-b7da-f1cd5afad678\") " Nov 02 23:00:23.445192 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:23.445172 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c75bf587-6f92-43c4-b7da-f1cd5afad678-maas-api-service-ca" pod "c75bf587-6f92-43c4-b7da-f1cd5afad678" (UID: "c75bf587-6f92-43c4-b7da-f1cd5afad678"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:00:23.445301 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:23.445277 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c75bf587-6f92-43c4-b7da-f1cd5afad678-maas-api-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:00:23.447051 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:23.447031 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c75bf587-6f92-43c4-b7da-f1cd5afad678-kube-api-access-bw62m" pod "c75bf587-6f92-43c4-b7da-f1cd5afad678" (UID: "c75bf587-6f92-43c4-b7da-f1cd5afad678"). InnerVolumeSpecName "kube-api-access-bw62m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:00:23.545966 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:23.545876 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw62m\" (UniqueName: \"kubernetes.io/projected/c75bf587-6f92-43c4-b7da-f1cd5afad678-kube-api-access-bw62m\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:00:23.597320 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:23.597283 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn" Nov 02 23:00:24.024463 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:24.024439 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn"] Nov 02 23:00:24.026077 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:00:24.026051 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2644cc75_0d2f_4a15_b7d9_c9faa7d57d1c.slice/crio-51767a128c2a6723acc7b0993a39a6df0700f1bd0e37a218da335aa34a61d96b WatchSource:0}: Error finding container 51767a128c2a6723acc7b0993a39a6df0700f1bd0e37a218da335aa34a61d96b: Status 404 returned error can't find the container with id 51767a128c2a6723acc7b0993a39a6df0700f1bd0e37a218da335aa34a61d96b Nov 02 23:00:24.099627 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:24.099595 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn" event={"ID":"2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c","Type":"ContainerStarted","Data":"51767a128c2a6723acc7b0993a39a6df0700f1bd0e37a218da335aa34a61d96b"} Nov 02 23:00:24.100962 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:24.100941 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-l5srw" event={"ID":"c75bf587-6f92-43c4-b7da-f1cd5afad678","Type":"ContainerDied","Data":"61131360e5a5f1d2037750c81308950648bd74a4c03c1640e0abe89bbb5a2a50"} Nov 02 23:00:24.101053 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:24.100983 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-l5srw" Nov 02 23:00:24.101053 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:24.100993 5727 scope.go:122] "RemoveContainer" containerID="a6da73c0bc4134866691d0552e1fbb0682ce42ce32c8e9e8c0d445d813669acf" Nov 02 23:00:24.264091 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:24.264004 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894340-l5srw"] Nov 02 23:00:24.268166 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:24.268143 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894340-l5srw"] Nov 02 23:00:25.707896 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:25.707864 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f779f4ae-67fccdfcc9-gt7n9"] Nov 02 23:00:25.708478 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:25.708457 5727 cpu_manager.go:414] "containerMap: removing container" podUID="c75bf587-6f92-43c4-b7da-f1cd5afad678" containerName="cleanup" Nov 02 23:00:25.708532 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:25.708481 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c75bf587-6f92-43c4-b7da-f1cd5afad678" containerName="cleanup" Nov 02 23:00:25.708532 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:25.708500 5727 cpu_manager.go:414] "containerMap: removing container" podUID="c75bf587-6f92-43c4-b7da-f1cd5afad678" containerName="cleanup" Nov 02 23:00:25.708532 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:25.708509 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c75bf587-6f92-43c4-b7da-f1cd5afad678" containerName="cleanup" Nov 02 23:00:25.708532 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:25.708523 5727 cpu_manager.go:414] "containerMap: removing container" podUID="c75bf587-6f92-43c4-b7da-f1cd5afad678" containerName="cleanup" Nov 02 23:00:25.708532 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:25.708528 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c75bf587-6f92-43c4-b7da-f1cd5afad678" containerName="cleanup" Nov 02 23:00:25.708741 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:25.708621 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="c75bf587-6f92-43c4-b7da-f1cd5afad678" containerName="cleanup" Nov 02 23:00:25.708741 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:25.708650 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="c75bf587-6f92-43c4-b7da-f1cd5afad678" containerName="cleanup" Nov 02 23:00:25.711981 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:25.711960 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f779f4ae-67fccdfcc9-gt7n9" Nov 02 23:00:25.766151 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:25.766100 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f779f4ae\"" Nov 02 23:00:25.767828 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:25.767782 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsbbm\" (UniqueName: \"kubernetes.io/projected/3ea4766f-c234-4032-bd30-36e9e9a9896a-kube-api-access-gsbbm\") pod \"payload-pre-processing-e2e-ait-f779f4ae-67fccdfcc9-gt7n9\" (UID: \"3ea4766f-c234-4032-bd30-36e9e9a9896a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f779f4ae-67fccdfcc9-gt7n9" Nov 02 23:00:25.767957 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:25.767901 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ea4766f-c234-4032-bd30-36e9e9a9896a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f779f4ae-67fccdfcc9-gt7n9\" (UID: \"3ea4766f-c234-4032-bd30-36e9e9a9896a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f779f4ae-67fccdfcc9-gt7n9" Nov 02 23:00:25.769899 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:25.769878 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f779f4ae-67fccdfcc9-gt7n9"] Nov 02 23:00:25.815641 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:25.815612 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f779f4ae-dockercfg-jvn5n\"" Nov 02 23:00:25.855664 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:25.855628 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f779f4ae-5775cbf6d9-6lzz5"] Nov 02 23:00:25.859311 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:25.859279 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="c75bf587-6f92-43c4-b7da-f1cd5afad678" containerName="cleanup" Nov 02 23:00:25.877261 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:25.877225 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ea4766f-c234-4032-bd30-36e9e9a9896a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f779f4ae-67fccdfcc9-gt7n9\" (UID: \"3ea4766f-c234-4032-bd30-36e9e9a9896a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f779f4ae-67fccdfcc9-gt7n9" Nov 02 23:00:25.877492 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:25.877464 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsbbm\" (UniqueName: \"kubernetes.io/projected/3ea4766f-c234-4032-bd30-36e9e9a9896a-kube-api-access-gsbbm\") pod \"payload-pre-processing-e2e-ait-f779f4ae-67fccdfcc9-gt7n9\" (UID: \"3ea4766f-c234-4032-bd30-36e9e9a9896a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f779f4ae-67fccdfcc9-gt7n9" Nov 02 23:00:25.878846 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:25.878784 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ea4766f-c234-4032-bd30-36e9e9a9896a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f779f4ae-67fccdfcc9-gt7n9\" (UID: \"3ea4766f-c234-4032-bd30-36e9e9a9896a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f779f4ae-67fccdfcc9-gt7n9" Nov 02 23:00:25.879709 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:25.879688 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f779f4ae-5775cbf6d9-6lzz5" Nov 02 23:00:25.898634 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:25.898604 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c75bf587-6f92-43c4-b7da-f1cd5afad678" path="/var/lib/kubelet/pods/c75bf587-6f92-43c4-b7da-f1cd5afad678/volumes" Nov 02 23:00:25.916303 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:25.916216 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f779f4ae-5775cbf6d9-6lzz5"] Nov 02 23:00:25.938823 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:25.938792 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsbbm\" (UniqueName: \"kubernetes.io/projected/3ea4766f-c234-4032-bd30-36e9e9a9896a-kube-api-access-gsbbm\") pod \"payload-pre-processing-e2e-ait-f779f4ae-67fccdfcc9-gt7n9\" (UID: \"3ea4766f-c234-4032-bd30-36e9e9a9896a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f779f4ae-67fccdfcc9-gt7n9" Nov 02 23:00:25.978942 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:25.978857 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe11484-ba7f-4b38-8a9c-1bf3893578a5-plugins-config-volume\") pod \"payload-processing-e2e-ait-f779f4ae-5775cbf6d9-6lzz5\" (UID: \"6fe11484-ba7f-4b38-8a9c-1bf3893578a5\") " pod="openshift-ingress/payload-processing-e2e-ait-f779f4ae-5775cbf6d9-6lzz5" Nov 02 23:00:25.979095 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:25.979031 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmn8x\" (UniqueName: \"kubernetes.io/projected/6fe11484-ba7f-4b38-8a9c-1bf3893578a5-kube-api-access-hmn8x\") pod \"payload-processing-e2e-ait-f779f4ae-5775cbf6d9-6lzz5\" (UID: \"6fe11484-ba7f-4b38-8a9c-1bf3893578a5\") " pod="openshift-ingress/payload-processing-e2e-ait-f779f4ae-5775cbf6d9-6lzz5" Nov 02 23:00:26.028303 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:26.028269 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f779f4ae-67fccdfcc9-gt7n9" Nov 02 23:00:26.082814 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:26.082773 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmn8x\" (UniqueName: \"kubernetes.io/projected/6fe11484-ba7f-4b38-8a9c-1bf3893578a5-kube-api-access-hmn8x\") pod \"payload-processing-e2e-ait-f779f4ae-5775cbf6d9-6lzz5\" (UID: \"6fe11484-ba7f-4b38-8a9c-1bf3893578a5\") " pod="openshift-ingress/payload-processing-e2e-ait-f779f4ae-5775cbf6d9-6lzz5" Nov 02 23:00:26.082989 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:26.082874 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe11484-ba7f-4b38-8a9c-1bf3893578a5-plugins-config-volume\") pod \"payload-processing-e2e-ait-f779f4ae-5775cbf6d9-6lzz5\" (UID: \"6fe11484-ba7f-4b38-8a9c-1bf3893578a5\") " pod="openshift-ingress/payload-processing-e2e-ait-f779f4ae-5775cbf6d9-6lzz5" Nov 02 23:00:26.083762 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:26.083738 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe11484-ba7f-4b38-8a9c-1bf3893578a5-plugins-config-volume\") pod \"payload-processing-e2e-ait-f779f4ae-5775cbf6d9-6lzz5\" (UID: \"6fe11484-ba7f-4b38-8a9c-1bf3893578a5\") " pod="openshift-ingress/payload-processing-e2e-ait-f779f4ae-5775cbf6d9-6lzz5" Nov 02 23:00:26.144042 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:26.144013 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmn8x\" (UniqueName: \"kubernetes.io/projected/6fe11484-ba7f-4b38-8a9c-1bf3893578a5-kube-api-access-hmn8x\") pod \"payload-processing-e2e-ait-f779f4ae-5775cbf6d9-6lzz5\" (UID: \"6fe11484-ba7f-4b38-8a9c-1bf3893578a5\") " pod="openshift-ingress/payload-processing-e2e-ait-f779f4ae-5775cbf6d9-6lzz5" Nov 02 23:00:26.198672 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:26.198651 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f779f4ae-5775cbf6d9-6lzz5" Nov 02 23:00:26.413580 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:26.413560 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f779f4ae-67fccdfcc9-gt7n9"] Nov 02 23:00:26.416105 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:00:26.416074 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea4766f_c234_4032_bd30_36e9e9a9896a.slice/crio-050220df6573ab244b5a9e84e431984f44dfa0991fb2e6136a73862cba7ca4a5 WatchSource:0}: Error finding container 050220df6573ab244b5a9e84e431984f44dfa0991fb2e6136a73862cba7ca4a5: Status 404 returned error can't find the container with id 050220df6573ab244b5a9e84e431984f44dfa0991fb2e6136a73862cba7ca4a5 Nov 02 23:00:26.572036 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:26.572009 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f779f4ae-5775cbf6d9-6lzz5"] Nov 02 23:00:26.573738 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:00:26.573711 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe11484_ba7f_4b38_8a9c_1bf3893578a5.slice/crio-c24f0b5f8d4e9f46b675e71a58fb2869006df55b1bba93d66add4c1cb2b36fb7 WatchSource:0}: Error finding container c24f0b5f8d4e9f46b675e71a58fb2869006df55b1bba93d66add4c1cb2b36fb7: Status 404 returned error can't find the container with id c24f0b5f8d4e9f46b675e71a58fb2869006df55b1bba93d66add4c1cb2b36fb7 Nov 02 23:00:27.127827 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:27.127781 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f779f4ae-67fccdfcc9-gt7n9" event={"ID":"3ea4766f-c234-4032-bd30-36e9e9a9896a","Type":"ContainerStarted","Data":"587ceadd0b84b0d9b32e3c62bd00cbe85d194ceffdc756359914db3dd86055d0"} Nov 02 23:00:27.128292 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:27.127847 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f779f4ae-67fccdfcc9-gt7n9" Nov 02 23:00:27.128292 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:27.127869 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f779f4ae-67fccdfcc9-gt7n9" event={"ID":"3ea4766f-c234-4032-bd30-36e9e9a9896a","Type":"ContainerStarted","Data":"050220df6573ab244b5a9e84e431984f44dfa0991fb2e6136a73862cba7ca4a5"} Nov 02 23:00:27.129463 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:27.129435 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn" event={"ID":"2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c","Type":"ContainerStarted","Data":"eed7ed19324d300d352163cbc638f21b7a1ee23e2c9a16e356b59c3663802c77"} Nov 02 23:00:27.129587 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:27.129574 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn" Nov 02 23:00:27.131067 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:27.131048 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f779f4ae-5775cbf6d9-6lzz5" event={"ID":"6fe11484-ba7f-4b38-8a9c-1bf3893578a5","Type":"ContainerStarted","Data":"715f6260402a9f6ae62646afe75f9ef733e202ee8b3cb06f9d0f8fec546132ff"} Nov 02 23:00:27.131181 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:27.131080 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f779f4ae-5775cbf6d9-6lzz5" event={"ID":"6fe11484-ba7f-4b38-8a9c-1bf3893578a5","Type":"ContainerStarted","Data":"c24f0b5f8d4e9f46b675e71a58fb2869006df55b1bba93d66add4c1cb2b36fb7"} Nov 02 23:00:27.131181 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:27.131098 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f779f4ae-5775cbf6d9-6lzz5" Nov 02 23:00:27.244010 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:27.243958 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f779f4ae-67fccdfcc9-gt7n9" podStartSLOduration=2.243943719 podStartE2EDuration="2.243943719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:00:27.242634784 +0000 UTC m=+713.481132464" watchObservedRunningTime="2026-11-02 23:00:27.243943719 +0000 UTC m=+713.482441380" Nov 02 23:00:27.367617 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:27.367561 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn" podStartSLOduration=3.254949088 podStartE2EDuration="5.367545385s" totalImagesPullingTime="2.112596297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:00:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:00:27.365028454 +0000 UTC m=+713.603526135" watchObservedRunningTime="2026-11-02 23:00:27.367545385 +0000 UTC m=+713.606043047" Nov 02 23:00:27.484349 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:27.484242 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f779f4ae-5775cbf6d9-6lzz5" podStartSLOduration=2.484221516 podStartE2EDuration="2.484221516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:00:27.481186036 +0000 UTC m=+713.719683720" watchObservedRunningTime="2026-11-02 23:00:27.484221516 +0000 UTC m=+713.722719179" Nov 02 23:00:27.845939 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:00:27.845906 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:00:28.192819 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:28.192744 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7d7bf49f-9c4677987-qvqw6"] Nov 02 23:00:28.198807 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:28.198789 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7bf49f-9c4677987-qvqw6" Nov 02 23:00:28.252439 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:28.252412 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7d7bf49f-dockercfg-8jxbd\"" Nov 02 23:00:28.255007 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:28.254981 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7d7bf49f-9c4677987-qvqw6"] Nov 02 23:00:28.301429 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:28.301404 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7d7bf49f\"" Nov 02 23:00:28.306083 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:28.306048 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b073ad5-b50e-4b8a-a750-9cbc29638e11-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7d7bf49f-9c4677987-qvqw6\" (UID: \"0b073ad5-b50e-4b8a-a750-9cbc29638e11\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7bf49f-9c4677987-qvqw6" Nov 02 23:00:28.306244 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:28.306214 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqwf9\" (UniqueName: \"kubernetes.io/projected/0b073ad5-b50e-4b8a-a750-9cbc29638e11-kube-api-access-wqwf9\") pod \"payload-pre-processing-e2e-ait-7d7bf49f-9c4677987-qvqw6\" (UID: \"0b073ad5-b50e-4b8a-a750-9cbc29638e11\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7bf49f-9c4677987-qvqw6" Nov 02 23:00:28.397602 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:28.397568 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7d7bf49f-574d8b8885-mwtww"] Nov 02 23:00:28.401738 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:28.401723 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7d7bf49f-574d8b8885-mwtww" Nov 02 23:00:28.407344 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:28.407321 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b073ad5-b50e-4b8a-a750-9cbc29638e11-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7d7bf49f-9c4677987-qvqw6\" (UID: \"0b073ad5-b50e-4b8a-a750-9cbc29638e11\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7bf49f-9c4677987-qvqw6" Nov 02 23:00:28.407447 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:28.407417 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqwf9\" (UniqueName: \"kubernetes.io/projected/0b073ad5-b50e-4b8a-a750-9cbc29638e11-kube-api-access-wqwf9\") pod \"payload-pre-processing-e2e-ait-7d7bf49f-9c4677987-qvqw6\" (UID: \"0b073ad5-b50e-4b8a-a750-9cbc29638e11\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7bf49f-9c4677987-qvqw6" Nov 02 23:00:28.408028 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:28.408005 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b073ad5-b50e-4b8a-a750-9cbc29638e11-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7d7bf49f-9c4677987-qvqw6\" (UID: \"0b073ad5-b50e-4b8a-a750-9cbc29638e11\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7bf49f-9c4677987-qvqw6" Nov 02 23:00:28.460543 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:28.460479 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7d7bf49f-574d8b8885-mwtww"] Nov 02 23:00:28.468877 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:28.468851 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqwf9\" (UniqueName: \"kubernetes.io/projected/0b073ad5-b50e-4b8a-a750-9cbc29638e11-kube-api-access-wqwf9\") pod \"payload-pre-processing-e2e-ait-7d7bf49f-9c4677987-qvqw6\" (UID: \"0b073ad5-b50e-4b8a-a750-9cbc29638e11\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7bf49f-9c4677987-qvqw6" Nov 02 23:00:28.508471 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:28.508434 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1241a260-7110-40dc-9f4b-99057b083a0e-plugins-config-volume\") pod \"payload-processing-e2e-ait-7d7bf49f-574d8b8885-mwtww\" (UID: \"1241a260-7110-40dc-9f4b-99057b083a0e\") " pod="openshift-ingress/payload-processing-e2e-ait-7d7bf49f-574d8b8885-mwtww" Nov 02 23:00:28.508666 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:28.508641 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6scq\" (UniqueName: \"kubernetes.io/projected/1241a260-7110-40dc-9f4b-99057b083a0e-kube-api-access-f6scq\") pod \"payload-processing-e2e-ait-7d7bf49f-574d8b8885-mwtww\" (UID: \"1241a260-7110-40dc-9f4b-99057b083a0e\") " pod="openshift-ingress/payload-processing-e2e-ait-7d7bf49f-574d8b8885-mwtww" Nov 02 23:00:28.513204 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:28.513177 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7bf49f-9c4677987-qvqw6" Nov 02 23:00:28.609895 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:28.609864 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6scq\" (UniqueName: \"kubernetes.io/projected/1241a260-7110-40dc-9f4b-99057b083a0e-kube-api-access-f6scq\") pod \"payload-processing-e2e-ait-7d7bf49f-574d8b8885-mwtww\" (UID: \"1241a260-7110-40dc-9f4b-99057b083a0e\") " pod="openshift-ingress/payload-processing-e2e-ait-7d7bf49f-574d8b8885-mwtww" Nov 02 23:00:28.610028 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:28.609964 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1241a260-7110-40dc-9f4b-99057b083a0e-plugins-config-volume\") pod \"payload-processing-e2e-ait-7d7bf49f-574d8b8885-mwtww\" (UID: \"1241a260-7110-40dc-9f4b-99057b083a0e\") " pod="openshift-ingress/payload-processing-e2e-ait-7d7bf49f-574d8b8885-mwtww" Nov 02 23:00:28.610629 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:28.610606 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1241a260-7110-40dc-9f4b-99057b083a0e-plugins-config-volume\") pod \"payload-processing-e2e-ait-7d7bf49f-574d8b8885-mwtww\" (UID: \"1241a260-7110-40dc-9f4b-99057b083a0e\") " pod="openshift-ingress/payload-processing-e2e-ait-7d7bf49f-574d8b8885-mwtww" Nov 02 23:00:28.670294 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:28.670271 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6scq\" (UniqueName: \"kubernetes.io/projected/1241a260-7110-40dc-9f4b-99057b083a0e-kube-api-access-f6scq\") pod \"payload-processing-e2e-ait-7d7bf49f-574d8b8885-mwtww\" (UID: \"1241a260-7110-40dc-9f4b-99057b083a0e\") " pod="openshift-ingress/payload-processing-e2e-ait-7d7bf49f-574d8b8885-mwtww" Nov 02 23:00:28.714715 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:28.714653 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7d7bf49f-574d8b8885-mwtww" Nov 02 23:00:28.916539 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:28.916347 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7d7bf49f-9c4677987-qvqw6"] Nov 02 23:00:28.918799 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:00:28.918774 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b073ad5_b50e_4b8a_a750_9cbc29638e11.slice/crio-aea809643766582fb10d250eeac12acdd71a0eca8a5d3f1a6d49578ec72b0bbe WatchSource:0}: Error finding container aea809643766582fb10d250eeac12acdd71a0eca8a5d3f1a6d49578ec72b0bbe: Status 404 returned error can't find the container with id aea809643766582fb10d250eeac12acdd71a0eca8a5d3f1a6d49578ec72b0bbe Nov 02 23:00:29.089622 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:29.089594 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7d7bf49f-574d8b8885-mwtww"] Nov 02 23:00:29.090629 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:00:29.090605 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1241a260_7110_40dc_9f4b_99057b083a0e.slice/crio-ceab358279fa549813b9027cc7640ee373ac7a9c3862655b241ed513b48d132e WatchSource:0}: Error finding container ceab358279fa549813b9027cc7640ee373ac7a9c3862655b241ed513b48d132e: Status 404 returned error can't find the container with id ceab358279fa549813b9027cc7640ee373ac7a9c3862655b241ed513b48d132e Nov 02 23:00:29.145458 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:29.145428 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7bf49f-9c4677987-qvqw6" event={"ID":"0b073ad5-b50e-4b8a-a750-9cbc29638e11","Type":"ContainerStarted","Data":"1bcafacc2a45cca32022b05a77e055d3f350584dd29442ec851963169f1cec50"} Nov 02 23:00:29.145572 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:29.145493 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7bf49f-9c4677987-qvqw6" event={"ID":"0b073ad5-b50e-4b8a-a750-9cbc29638e11","Type":"ContainerStarted","Data":"aea809643766582fb10d250eeac12acdd71a0eca8a5d3f1a6d49578ec72b0bbe"} Nov 02 23:00:29.145633 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:29.145578 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7bf49f-9c4677987-qvqw6" Nov 02 23:00:29.147179 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:29.147156 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7d7bf49f-574d8b8885-mwtww" event={"ID":"1241a260-7110-40dc-9f4b-99057b083a0e","Type":"ContainerStarted","Data":"ceab358279fa549813b9027cc7640ee373ac7a9c3862655b241ed513b48d132e"} Nov 02 23:00:29.262451 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:29.262356 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7bf49f-9c4677987-qvqw6" podStartSLOduration=1.262343368 podStartE2EDuration="1.262343368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:00:29.259829127 +0000 UTC m=+715.498326803" watchObservedRunningTime="2026-11-02 23:00:29.262343368 +0000 UTC m=+715.500841028" Nov 02 23:00:30.154200 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:30.154166 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7d7bf49f-574d8b8885-mwtww" event={"ID":"1241a260-7110-40dc-9f4b-99057b083a0e","Type":"ContainerStarted","Data":"b87816f868a8a8a9dc15d5727315c6f0cc68d6299faec1f06468918d82af3844"} Nov 02 23:00:30.154395 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:30.154342 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-7d7bf49f-574d8b8885-mwtww" Nov 02 23:00:30.273190 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:30.273110 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-7d7bf49f-574d8b8885-mwtww" podStartSLOduration=2.273097115 podStartE2EDuration="2.273097115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:00:30.271665159 +0000 UTC m=+716.510162844" watchObservedRunningTime="2026-11-02 23:00:30.273097115 +0000 UTC m=+716.511594808" Nov 02 23:00:30.865314 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:30.865285 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5dd28258-8478955cbc-s5k87"] Nov 02 23:00:30.869185 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:30.869170 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5dd28258-8478955cbc-s5k87" Nov 02 23:00:30.923348 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:30.922906 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5dd28258\"" Nov 02 23:00:30.923544 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:30.923000 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5dd28258-dockercfg-p2drx\"" Nov 02 23:00:30.927036 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:30.927009 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5dd28258-8478955cbc-s5k87"] Nov 02 23:00:30.933993 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:30.933964 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmmbb\" (UniqueName: \"kubernetes.io/projected/070d859c-d644-4b85-9350-e1ce6c37d1aa-kube-api-access-vmmbb\") pod \"payload-pre-processing-e2e-ait-5dd28258-8478955cbc-s5k87\" (UID: \"070d859c-d644-4b85-9350-e1ce6c37d1aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5dd28258-8478955cbc-s5k87" Nov 02 23:00:30.934240 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:30.934140 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/070d859c-d644-4b85-9350-e1ce6c37d1aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5dd28258-8478955cbc-s5k87\" (UID: \"070d859c-d644-4b85-9350-e1ce6c37d1aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5dd28258-8478955cbc-s5k87" Nov 02 23:00:31.035320 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:31.035291 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/070d859c-d644-4b85-9350-e1ce6c37d1aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5dd28258-8478955cbc-s5k87\" (UID: \"070d859c-d644-4b85-9350-e1ce6c37d1aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5dd28258-8478955cbc-s5k87" Nov 02 23:00:31.035467 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:31.035349 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmmbb\" (UniqueName: \"kubernetes.io/projected/070d859c-d644-4b85-9350-e1ce6c37d1aa-kube-api-access-vmmbb\") pod \"payload-pre-processing-e2e-ait-5dd28258-8478955cbc-s5k87\" (UID: \"070d859c-d644-4b85-9350-e1ce6c37d1aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5dd28258-8478955cbc-s5k87" Nov 02 23:00:31.036080 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:31.036060 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/070d859c-d644-4b85-9350-e1ce6c37d1aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5dd28258-8478955cbc-s5k87\" (UID: \"070d859c-d644-4b85-9350-e1ce6c37d1aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5dd28258-8478955cbc-s5k87" Nov 02 23:00:31.096581 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:31.096544 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmmbb\" (UniqueName: \"kubernetes.io/projected/070d859c-d644-4b85-9350-e1ce6c37d1aa-kube-api-access-vmmbb\") pod \"payload-pre-processing-e2e-ait-5dd28258-8478955cbc-s5k87\" (UID: \"070d859c-d644-4b85-9350-e1ce6c37d1aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5dd28258-8478955cbc-s5k87" Nov 02 23:00:31.183005 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:31.182933 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5dd28258-8478955cbc-s5k87" Nov 02 23:00:31.558233 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:31.558205 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5dd28258-8478955cbc-s5k87"] Nov 02 23:00:31.559641 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:00:31.559611 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070d859c_d644_4b85_9350_e1ce6c37d1aa.slice/crio-e370faea54404192ea0ff1abba7c16dfd14e9c5c8e6b60c99af3bdf73c1a2b11 WatchSource:0}: Error finding container e370faea54404192ea0ff1abba7c16dfd14e9c5c8e6b60c99af3bdf73c1a2b11: Status 404 returned error can't find the container with id e370faea54404192ea0ff1abba7c16dfd14e9c5c8e6b60c99af3bdf73c1a2b11 Nov 02 23:00:32.168499 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:32.168467 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5dd28258-8478955cbc-s5k87" event={"ID":"070d859c-d644-4b85-9350-e1ce6c37d1aa","Type":"ContainerStarted","Data":"f904352cb3979a913b67cc7afb4ad5bba855c6bf8380d6d03723c5c750c31624"} Nov 02 23:00:32.168692 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:32.168668 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5dd28258-8478955cbc-s5k87" Nov 02 23:00:32.168749 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:32.168704 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5dd28258-8478955cbc-s5k87" event={"ID":"070d859c-d644-4b85-9350-e1ce6c37d1aa","Type":"ContainerStarted","Data":"e370faea54404192ea0ff1abba7c16dfd14e9c5c8e6b60c99af3bdf73c1a2b11"} Nov 02 23:00:32.286284 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:32.286231 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5dd28258-8478955cbc-s5k87" podStartSLOduration=2.286218873 podStartE2EDuration="2.286218873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:00:32.283471685 +0000 UTC m=+718.521969347" watchObservedRunningTime="2026-11-02 23:00:32.286218873 +0000 UTC m=+718.524716534" Nov 02 23:00:33.096659 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:33.096608 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-fed9e8ee-787876d555-j2dhc" Nov 02 23:00:33.097056 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:33.096709 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed9e8ee-644dc48579-kvq2w" Nov 02 23:00:33.141620 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:33.141594 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn" Nov 02 23:00:38.139728 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:38.139695 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f779f4ae-5775cbf6d9-6lzz5" Nov 02 23:00:38.140249 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:38.139773 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f779f4ae-67fccdfcc9-gt7n9" Nov 02 23:00:40.158263 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:40.158240 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7bf49f-9c4677987-qvqw6" Nov 02 23:00:41.164054 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:41.164006 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-7d7bf49f-574d8b8885-mwtww" Nov 02 23:00:43.178782 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:43.178753 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5dd28258-8478955cbc-s5k87" Nov 02 23:00:43.944313 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:43.944276 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97"] Nov 02 23:00:43.949486 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:43.949469 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97" Nov 02 23:00:44.004271 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:44.004239 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f779f4ae-serving-cert\"" Nov 02 23:00:44.056581 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:44.056556 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97"] Nov 02 23:00:44.076188 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:44.076111 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa69d14-76a5-471d-b452-bbbacf892687-openshift-service-ca\") pod \"maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97\" (UID: \"3fa69d14-76a5-471d-b452-bbbacf892687\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97" Nov 02 23:00:44.076381 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:44.076192 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4scg\" (UniqueName: \"kubernetes.io/projected/3fa69d14-76a5-471d-b452-bbbacf892687-kube-api-access-d4scg\") pod \"maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97\" (UID: \"3fa69d14-76a5-471d-b452-bbbacf892687\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97" Nov 02 23:00:44.076381 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:44.076277 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fa69d14-76a5-471d-b452-bbbacf892687-maas-api-tls\") pod \"maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97\" (UID: \"3fa69d14-76a5-471d-b452-bbbacf892687\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97" Nov 02 23:00:44.076537 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:44.076450 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fa69d14-76a5-471d-b452-bbbacf892687-metrics-certs\") pod \"maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97\" (UID: \"3fa69d14-76a5-471d-b452-bbbacf892687\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97" Nov 02 23:00:44.177818 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:44.177785 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa69d14-76a5-471d-b452-bbbacf892687-openshift-service-ca\") pod \"maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97\" (UID: \"3fa69d14-76a5-471d-b452-bbbacf892687\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97" Nov 02 23:00:44.177818 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:44.177817 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4scg\" (UniqueName: \"kubernetes.io/projected/3fa69d14-76a5-471d-b452-bbbacf892687-kube-api-access-d4scg\") pod \"maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97\" (UID: \"3fa69d14-76a5-471d-b452-bbbacf892687\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97" Nov 02 23:00:44.178039 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:44.177868 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fa69d14-76a5-471d-b452-bbbacf892687-maas-api-tls\") pod \"maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97\" (UID: \"3fa69d14-76a5-471d-b452-bbbacf892687\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97" Nov 02 23:00:44.178039 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:44.177894 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fa69d14-76a5-471d-b452-bbbacf892687-metrics-certs\") pod \"maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97\" (UID: \"3fa69d14-76a5-471d-b452-bbbacf892687\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97" Nov 02 23:00:44.178453 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:44.178432 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa69d14-76a5-471d-b452-bbbacf892687-openshift-service-ca\") pod \"maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97\" (UID: \"3fa69d14-76a5-471d-b452-bbbacf892687\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97" Nov 02 23:00:44.180650 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:44.180628 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fa69d14-76a5-471d-b452-bbbacf892687-maas-api-tls\") pod \"maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97\" (UID: \"3fa69d14-76a5-471d-b452-bbbacf892687\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97" Nov 02 23:00:44.181032 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:44.180632 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fa69d14-76a5-471d-b452-bbbacf892687-metrics-certs\") pod \"maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97\" (UID: \"3fa69d14-76a5-471d-b452-bbbacf892687\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97" Nov 02 23:00:44.239381 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:44.239320 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4scg\" (UniqueName: \"kubernetes.io/projected/3fa69d14-76a5-471d-b452-bbbacf892687-kube-api-access-d4scg\") pod \"maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97\" (UID: \"3fa69d14-76a5-471d-b452-bbbacf892687\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97" Nov 02 23:00:44.263054 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:44.263031 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97" Nov 02 23:00:44.641964 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:44.641939 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97"] Nov 02 23:00:44.643797 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:00:44.643766 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa69d14_76a5_471d_b452_bbbacf892687.slice/crio-589986e9e493a790aea58d30efa5854c4d0d668e6dcddd00c39d7bddd32bce76 WatchSource:0}: Error finding container 589986e9e493a790aea58d30efa5854c4d0d668e6dcddd00c39d7bddd32bce76: Status 404 returned error can't find the container with id 589986e9e493a790aea58d30efa5854c4d0d668e6dcddd00c39d7bddd32bce76 Nov 02 23:00:45.267694 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:45.267656 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97" event={"ID":"3fa69d14-76a5-471d-b452-bbbacf892687","Type":"ContainerStarted","Data":"589986e9e493a790aea58d30efa5854c4d0d668e6dcddd00c39d7bddd32bce76"} Nov 02 23:00:46.276313 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:46.276275 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97" event={"ID":"3fa69d14-76a5-471d-b452-bbbacf892687","Type":"ContainerStarted","Data":"0ceeda311ed8dac240dfc22d5dfd25408c9583139d977e31d2cd6adcc39ad0cd"} Nov 02 23:00:46.276720 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:46.276367 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97" Nov 02 23:00:46.399401 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:46.399352 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97" podStartSLOduration=2.6345750900000002 podStartE2EDuration="3.399337603s" totalImagesPullingTime="764.762513ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:00:46.399173787 +0000 UTC m=+732.637671466" watchObservedRunningTime="2026-11-02 23:00:46.399337603 +0000 UTC m=+732.637835260" Nov 02 23:00:49.212326 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:49.212287 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq"] Nov 02 23:00:49.216208 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:49.216188 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq" Nov 02 23:00:49.320008 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:49.319969 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-708f46a0-serving-cert\"" Nov 02 23:00:49.322763 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:49.322744 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq"] Nov 02 23:00:49.329728 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:49.329711 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73d19747-5f1e-459c-81f7-cf4eefc86b40-maas-api-tls\") pod \"maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq\" (UID: \"73d19747-5f1e-459c-81f7-cf4eefc86b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq" Nov 02 23:00:49.329811 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:49.329742 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d19747-5f1e-459c-81f7-cf4eefc86b40-openshift-service-ca\") pod \"maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq\" (UID: \"73d19747-5f1e-459c-81f7-cf4eefc86b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq" Nov 02 23:00:49.329864 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:49.329828 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73d19747-5f1e-459c-81f7-cf4eefc86b40-metrics-certs\") pod \"maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq\" (UID: \"73d19747-5f1e-459c-81f7-cf4eefc86b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq" Nov 02 23:00:49.329970 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:49.329950 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kdz2\" (UniqueName: \"kubernetes.io/projected/73d19747-5f1e-459c-81f7-cf4eefc86b40-kube-api-access-9kdz2\") pod \"maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq\" (UID: \"73d19747-5f1e-459c-81f7-cf4eefc86b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq" Nov 02 23:00:49.430772 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:49.430738 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73d19747-5f1e-459c-81f7-cf4eefc86b40-metrics-certs\") pod \"maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq\" (UID: \"73d19747-5f1e-459c-81f7-cf4eefc86b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq" Nov 02 23:00:49.430980 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:49.430817 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kdz2\" (UniqueName: \"kubernetes.io/projected/73d19747-5f1e-459c-81f7-cf4eefc86b40-kube-api-access-9kdz2\") pod \"maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq\" (UID: \"73d19747-5f1e-459c-81f7-cf4eefc86b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq" Nov 02 23:00:49.430980 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:49.430875 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73d19747-5f1e-459c-81f7-cf4eefc86b40-maas-api-tls\") pod \"maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq\" (UID: \"73d19747-5f1e-459c-81f7-cf4eefc86b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq" Nov 02 23:00:49.430980 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:49.430907 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d19747-5f1e-459c-81f7-cf4eefc86b40-openshift-service-ca\") pod \"maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq\" (UID: \"73d19747-5f1e-459c-81f7-cf4eefc86b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq" Nov 02 23:00:49.431196 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:00:49.431049 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-708f46a0-serving-cert: secret "maas-api-e2e-derive-708f46a0-serving-cert" not found Nov 02 23:00:49.431196 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:00:49.431169 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73d19747-5f1e-459c-81f7-cf4eefc86b40-maas-api-tls podName:73d19747-5f1e-459c-81f7-cf4eefc86b40 nodeName:}" failed. No retries permitted until 2026-11-02 23:00:49.931104762 +0000 UTC m=+736.169602414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/73d19747-5f1e-459c-81f7-cf4eefc86b40-maas-api-tls") pod "maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq" (UID: "73d19747-5f1e-459c-81f7-cf4eefc86b40") : secret "maas-api-e2e-derive-708f46a0-serving-cert" not found Nov 02 23:00:49.431630 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:49.431607 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d19747-5f1e-459c-81f7-cf4eefc86b40-openshift-service-ca\") pod \"maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq\" (UID: \"73d19747-5f1e-459c-81f7-cf4eefc86b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq" Nov 02 23:00:49.433409 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:49.433391 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73d19747-5f1e-459c-81f7-cf4eefc86b40-metrics-certs\") pod \"maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq\" (UID: \"73d19747-5f1e-459c-81f7-cf4eefc86b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq" Nov 02 23:00:49.491699 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:49.491628 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kdz2\" (UniqueName: \"kubernetes.io/projected/73d19747-5f1e-459c-81f7-cf4eefc86b40-kube-api-access-9kdz2\") pod \"maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq\" (UID: \"73d19747-5f1e-459c-81f7-cf4eefc86b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq" Nov 02 23:00:49.936956 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:49.936920 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73d19747-5f1e-459c-81f7-cf4eefc86b40-maas-api-tls\") pod \"maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq\" (UID: \"73d19747-5f1e-459c-81f7-cf4eefc86b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq" Nov 02 23:00:49.939426 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:49.939402 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73d19747-5f1e-459c-81f7-cf4eefc86b40-maas-api-tls\") pod \"maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq\" (UID: \"73d19747-5f1e-459c-81f7-cf4eefc86b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq" Nov 02 23:00:50.128950 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:50.128919 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq" Nov 02 23:00:50.509684 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:50.509655 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq"] Nov 02 23:00:50.511208 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:00:50.511181 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73d19747_5f1e_459c_81f7_cf4eefc86b40.slice/crio-1c47eaf6a9b4996418491eccb0364e66014bd965d6abd9d0668c115b83c77e92 WatchSource:0}: Error finding container 1c47eaf6a9b4996418491eccb0364e66014bd965d6abd9d0668c115b83c77e92: Status 404 returned error can't find the container with id 1c47eaf6a9b4996418491eccb0364e66014bd965d6abd9d0668c115b83c77e92 Nov 02 23:00:51.310121 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:51.310074 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq" event={"ID":"73d19747-5f1e-459c-81f7-cf4eefc86b40","Type":"ContainerStarted","Data":"1c47eaf6a9b4996418491eccb0364e66014bd965d6abd9d0668c115b83c77e92"} Nov 02 23:00:52.288488 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:52.288458 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97" Nov 02 23:00:52.317107 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:52.317078 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq" event={"ID":"73d19747-5f1e-459c-81f7-cf4eefc86b40","Type":"ContainerStarted","Data":"d2c71e006d37c8cb90a6cffc2c9acd3f23e406bea8edc19df2b172149aa153ad"} Nov 02 23:00:52.317251 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:52.317169 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq" Nov 02 23:00:52.524581 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:52.524525 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq" podStartSLOduration=2.708884281 podStartE2EDuration="3.524510428s" totalImagesPullingTime="815.626147ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:00:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:00:52.522021678 +0000 UTC m=+738.760519376" watchObservedRunningTime="2026-11-02 23:00:52.524510428 +0000 UTC m=+738.763008084" Nov 02 23:00:57.844586 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:00:57.844544 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:00:58.328772 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:00:58.328739 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq" Nov 02 23:01:13.354435 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:13.354397 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7d7bf49f-574d8b8885-mwtww"] Nov 02 23:01:13.464373 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:13.464339 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7d7bf49f-9c4677987-qvqw6"] Nov 02 23:01:18.355527 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:18.355457 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-7d7bf49f-574d8b8885-mwtww" podUID="1241a260-7110-40dc-9f4b-99057b083a0e" containerName="payload-processing" containerID="cri-o://b87816f868a8a8a9dc15d5727315c6f0cc68d6299faec1f06468918d82af3844" gracePeriod=25 Nov 02 23:01:18.465442 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:18.465400 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7bf49f-9c4677987-qvqw6" podUID="0b073ad5-b50e-4b8a-a750-9cbc29638e11" containerName="payload-pre-processing" containerID="cri-o://1bcafacc2a45cca32022b05a77e055d3f350584dd29442ec851963169f1cec50" gracePeriod=25 Nov 02 23:01:18.479689 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:18.479661 5727 generic.go:356] "Generic (PLEG): container finished" podID="1241a260-7110-40dc-9f4b-99057b083a0e" containerID="b87816f868a8a8a9dc15d5727315c6f0cc68d6299faec1f06468918d82af3844" exitCode=0 Nov 02 23:01:18.479805 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:18.479738 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7d7bf49f-574d8b8885-mwtww" event={"ID":"1241a260-7110-40dc-9f4b-99057b083a0e","Type":"ContainerDied","Data":"b87816f868a8a8a9dc15d5727315c6f0cc68d6299faec1f06468918d82af3844"} Nov 02 23:01:18.610938 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:18.610879 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7d7bf49f-574d8b8885-mwtww" Nov 02 23:01:18.736537 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:18.736516 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7bf49f-9c4677987-qvqw6" Nov 02 23:01:18.737194 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:18.737177 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1241a260-7110-40dc-9f4b-99057b083a0e-kube-api-access-f6scq\" (UniqueName: \"kubernetes.io/projected/1241a260-7110-40dc-9f4b-99057b083a0e-kube-api-access-f6scq\") pod \"1241a260-7110-40dc-9f4b-99057b083a0e\" (UID: \"1241a260-7110-40dc-9f4b-99057b083a0e\") " Nov 02 23:01:18.737282 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:18.737269 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1241a260-7110-40dc-9f4b-99057b083a0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1241a260-7110-40dc-9f4b-99057b083a0e-plugins-config-volume\") pod \"1241a260-7110-40dc-9f4b-99057b083a0e\" (UID: \"1241a260-7110-40dc-9f4b-99057b083a0e\") " Nov 02 23:01:18.737667 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:18.737639 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1241a260-7110-40dc-9f4b-99057b083a0e-plugins-config-volume" pod "1241a260-7110-40dc-9f4b-99057b083a0e" (UID: "1241a260-7110-40dc-9f4b-99057b083a0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:01:18.739472 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:18.739444 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1241a260-7110-40dc-9f4b-99057b083a0e-kube-api-access-f6scq" pod "1241a260-7110-40dc-9f4b-99057b083a0e" (UID: "1241a260-7110-40dc-9f4b-99057b083a0e"). InnerVolumeSpecName "kube-api-access-f6scq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:01:18.838052 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:18.838024 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b073ad5-b50e-4b8a-a750-9cbc29638e11-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b073ad5-b50e-4b8a-a750-9cbc29638e11-plugins-config-volume\") pod \"0b073ad5-b50e-4b8a-a750-9cbc29638e11\" (UID: \"0b073ad5-b50e-4b8a-a750-9cbc29638e11\") " Nov 02 23:01:18.838206 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:18.838090 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b073ad5-b50e-4b8a-a750-9cbc29638e11-kube-api-access-wqwf9\" (UniqueName: \"kubernetes.io/projected/0b073ad5-b50e-4b8a-a750-9cbc29638e11-kube-api-access-wqwf9\") pod \"0b073ad5-b50e-4b8a-a750-9cbc29638e11\" (UID: \"0b073ad5-b50e-4b8a-a750-9cbc29638e11\") " Nov 02 23:01:18.838329 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:18.838316 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1241a260-7110-40dc-9f4b-99057b083a0e-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:18.838366 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:18.838332 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6scq\" (UniqueName: \"kubernetes.io/projected/1241a260-7110-40dc-9f4b-99057b083a0e-kube-api-access-f6scq\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:18.838399 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:18.838382 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b073ad5-b50e-4b8a-a750-9cbc29638e11-plugins-config-volume" pod "0b073ad5-b50e-4b8a-a750-9cbc29638e11" (UID: "0b073ad5-b50e-4b8a-a750-9cbc29638e11"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:01:18.840249 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:18.840226 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b073ad5-b50e-4b8a-a750-9cbc29638e11-kube-api-access-wqwf9" pod "0b073ad5-b50e-4b8a-a750-9cbc29638e11" (UID: "0b073ad5-b50e-4b8a-a750-9cbc29638e11"). InnerVolumeSpecName "kube-api-access-wqwf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:01:18.939458 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:18.939383 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b073ad5-b50e-4b8a-a750-9cbc29638e11-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:18.939458 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:18.939409 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqwf9\" (UniqueName: \"kubernetes.io/projected/0b073ad5-b50e-4b8a-a750-9cbc29638e11-kube-api-access-wqwf9\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:19.487342 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:19.487304 5727 generic.go:356] "Generic (PLEG): container finished" podID="0b073ad5-b50e-4b8a-a750-9cbc29638e11" containerID="1bcafacc2a45cca32022b05a77e055d3f350584dd29442ec851963169f1cec50" exitCode=0 Nov 02 23:01:19.487790 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:19.487357 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7bf49f-9c4677987-qvqw6" Nov 02 23:01:19.487790 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:19.487378 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7bf49f-9c4677987-qvqw6" event={"ID":"0b073ad5-b50e-4b8a-a750-9cbc29638e11","Type":"ContainerDied","Data":"1bcafacc2a45cca32022b05a77e055d3f350584dd29442ec851963169f1cec50"} Nov 02 23:01:19.487790 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:19.487435 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7bf49f-9c4677987-qvqw6" event={"ID":"0b073ad5-b50e-4b8a-a750-9cbc29638e11","Type":"ContainerDied","Data":"aea809643766582fb10d250eeac12acdd71a0eca8a5d3f1a6d49578ec72b0bbe"} Nov 02 23:01:19.487790 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:19.487463 5727 scope.go:122] "RemoveContainer" containerID="1bcafacc2a45cca32022b05a77e055d3f350584dd29442ec851963169f1cec50" Nov 02 23:01:19.489106 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:19.489089 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7d7bf49f-574d8b8885-mwtww" Nov 02 23:01:19.489217 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:19.489195 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7d7bf49f-574d8b8885-mwtww" event={"ID":"1241a260-7110-40dc-9f4b-99057b083a0e","Type":"ContainerDied","Data":"ceab358279fa549813b9027cc7640ee373ac7a9c3862655b241ed513b48d132e"} Nov 02 23:01:19.497586 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:19.497565 5727 scope.go:122] "RemoveContainer" containerID="1bcafacc2a45cca32022b05a77e055d3f350584dd29442ec851963169f1cec50" Nov 02 23:01:19.497917 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:01:19.497893 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bcafacc2a45cca32022b05a77e055d3f350584dd29442ec851963169f1cec50\": container with ID starting with 1bcafacc2a45cca32022b05a77e055d3f350584dd29442ec851963169f1cec50 not found: ID does not exist" containerID="1bcafacc2a45cca32022b05a77e055d3f350584dd29442ec851963169f1cec50" Nov 02 23:01:19.497992 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:19.497929 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bcafacc2a45cca32022b05a77e055d3f350584dd29442ec851963169f1cec50"} err="failed to get container status \"1bcafacc2a45cca32022b05a77e055d3f350584dd29442ec851963169f1cec50\": rpc error: code = NotFound desc = could not find container \"1bcafacc2a45cca32022b05a77e055d3f350584dd29442ec851963169f1cec50\": container with ID starting with 1bcafacc2a45cca32022b05a77e055d3f350584dd29442ec851963169f1cec50 not found: ID does not exist" Nov 02 23:01:19.497992 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:19.497954 5727 scope.go:122] "RemoveContainer" containerID="b87816f868a8a8a9dc15d5727315c6f0cc68d6299faec1f06468918d82af3844" Nov 02 23:01:19.651616 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:19.651539 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7d7bf49f-574d8b8885-mwtww"] Nov 02 23:01:19.701199 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:19.701162 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7d7bf49f-574d8b8885-mwtww"] Nov 02 23:01:19.763552 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:19.763519 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7d7bf49f-9c4677987-qvqw6"] Nov 02 23:01:19.768326 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:19.768302 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7d7bf49f-9c4677987-qvqw6"] Nov 02 23:01:19.882978 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:19.882833 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b073ad5-b50e-4b8a-a750-9cbc29638e11" path="/var/lib/kubelet/pods/0b073ad5-b50e-4b8a-a750-9cbc29638e11/volumes" Nov 02 23:01:19.901053 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:19.883327 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1241a260-7110-40dc-9f4b-99057b083a0e" path="/var/lib/kubelet/pods/1241a260-7110-40dc-9f4b-99057b083a0e/volumes" Nov 02 23:01:22.966371 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:22.966341 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5dd28258-8478955cbc-s5k87"] Nov 02 23:01:25.306026 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:25.305986 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fed9e8ee-787876d555-j2dhc"] Nov 02 23:01:25.412170 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:25.412114 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fed9e8ee-644dc48579-kvq2w"] Nov 02 23:01:25.851875 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:25.851807 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn"] Nov 02 23:01:25.852121 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:25.852082 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn" podUID="2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c" containerName="maas-api" containerID="cri-o://eed7ed19324d300d352163cbc638f21b7a1ee23e2c9a16e356b59c3663802c77" gracePeriod=30 Nov 02 23:01:26.124613 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:26.124536 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn" Nov 02 23:01:26.213391 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:26.213359 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c-kube-api-access-8mwv5\" (UniqueName: \"kubernetes.io/projected/2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c-kube-api-access-8mwv5\") pod \"2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c\" (UID: \"2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c\") " Nov 02 23:01:26.213619 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:26.213506 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c-openshift-service-ca\") pod \"2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c\" (UID: \"2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c\") " Nov 02 23:01:26.213672 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:26.213625 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c-maas-api-tls\") pod \"2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c\" (UID: \"2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c\") " Nov 02 23:01:26.213672 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:26.213658 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c-metrics-certs\") pod \"2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c\" (UID: \"2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c\") " Nov 02 23:01:26.213960 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:26.213921 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c-openshift-service-ca" pod "2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c" (UID: "2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:01:26.215745 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:26.215720 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c-kube-api-access-8mwv5" pod "2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c" (UID: "2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c"). InnerVolumeSpecName "kube-api-access-8mwv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:01:26.215745 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:26.215730 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c-maas-api-tls" pod "2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c" (UID: "2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:01:26.215913 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:26.215787 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c-metrics-certs" pod "2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c" (UID: "2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:01:26.315223 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:26.315186 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:26.315223 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:26.315219 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:26.315223 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:26.315230 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:26.315677 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:26.315239 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mwv5\" (UniqueName: \"kubernetes.io/projected/2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c-kube-api-access-8mwv5\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:26.537354 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:26.537269 5727 generic.go:356] "Generic (PLEG): container finished" podID="2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c" containerID="eed7ed19324d300d352163cbc638f21b7a1ee23e2c9a16e356b59c3663802c77" exitCode=0 Nov 02 23:01:26.537354 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:26.537316 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn" Nov 02 23:01:26.537354 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:26.537340 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn" event={"ID":"2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c","Type":"ContainerDied","Data":"eed7ed19324d300d352163cbc638f21b7a1ee23e2c9a16e356b59c3663802c77"} Nov 02 23:01:26.537563 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:26.537387 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn" event={"ID":"2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c","Type":"ContainerDied","Data":"51767a128c2a6723acc7b0993a39a6df0700f1bd0e37a218da335aa34a61d96b"} Nov 02 23:01:26.537563 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:26.537411 5727 scope.go:122] "RemoveContainer" containerID="eed7ed19324d300d352163cbc638f21b7a1ee23e2c9a16e356b59c3663802c77" Nov 02 23:01:26.547830 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:26.547810 5727 scope.go:122] "RemoveContainer" containerID="eed7ed19324d300d352163cbc638f21b7a1ee23e2c9a16e356b59c3663802c77" Nov 02 23:01:26.548099 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:01:26.548076 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eed7ed19324d300d352163cbc638f21b7a1ee23e2c9a16e356b59c3663802c77\": container with ID starting with eed7ed19324d300d352163cbc638f21b7a1ee23e2c9a16e356b59c3663802c77 not found: ID does not exist" containerID="eed7ed19324d300d352163cbc638f21b7a1ee23e2c9a16e356b59c3663802c77" Nov 02 23:01:26.548214 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:26.548107 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eed7ed19324d300d352163cbc638f21b7a1ee23e2c9a16e356b59c3663802c77"} err="failed to get container status \"eed7ed19324d300d352163cbc638f21b7a1ee23e2c9a16e356b59c3663802c77\": rpc error: code = NotFound desc = could not find container \"eed7ed19324d300d352163cbc638f21b7a1ee23e2c9a16e356b59c3663802c77\": container with ID starting with eed7ed19324d300d352163cbc638f21b7a1ee23e2c9a16e356b59c3663802c77 not found: ID does not exist" Nov 02 23:01:26.701636 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:26.701590 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn"] Nov 02 23:01:26.705319 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:26.705287 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7d7bf49f-5674c545db-tmcbn"] Nov 02 23:01:27.846297 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:01:27.846258 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:01:27.881579 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:27.881549 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c" path="/var/lib/kubelet/pods/2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c/volumes" Nov 02 23:01:27.967934 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:27.967893 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5dd28258-8478955cbc-s5k87" podUID="070d859c-d644-4b85-9350-e1ce6c37d1aa" containerName="payload-pre-processing" containerID="cri-o://f904352cb3979a913b67cc7afb4ad5bba855c6bf8380d6d03723c5c750c31624" gracePeriod=25 Nov 02 23:01:28.235066 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:28.235039 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5dd28258-8478955cbc-s5k87" Nov 02 23:01:28.336694 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:28.336662 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/070d859c-d644-4b85-9350-e1ce6c37d1aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/070d859c-d644-4b85-9350-e1ce6c37d1aa-plugins-config-volume\") pod \"070d859c-d644-4b85-9350-e1ce6c37d1aa\" (UID: \"070d859c-d644-4b85-9350-e1ce6c37d1aa\") " Nov 02 23:01:28.336868 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:28.336738 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/070d859c-d644-4b85-9350-e1ce6c37d1aa-kube-api-access-vmmbb\" (UniqueName: \"kubernetes.io/projected/070d859c-d644-4b85-9350-e1ce6c37d1aa-kube-api-access-vmmbb\") pod \"070d859c-d644-4b85-9350-e1ce6c37d1aa\" (UID: \"070d859c-d644-4b85-9350-e1ce6c37d1aa\") " Nov 02 23:01:28.337096 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:28.337070 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/070d859c-d644-4b85-9350-e1ce6c37d1aa-plugins-config-volume" pod "070d859c-d644-4b85-9350-e1ce6c37d1aa" (UID: "070d859c-d644-4b85-9350-e1ce6c37d1aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:01:28.338993 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:28.338971 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/070d859c-d644-4b85-9350-e1ce6c37d1aa-kube-api-access-vmmbb" pod "070d859c-d644-4b85-9350-e1ce6c37d1aa" (UID: "070d859c-d644-4b85-9350-e1ce6c37d1aa"). InnerVolumeSpecName "kube-api-access-vmmbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:01:28.438471 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:28.438387 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/070d859c-d644-4b85-9350-e1ce6c37d1aa-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:28.438471 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:28.438416 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmmbb\" (UniqueName: \"kubernetes.io/projected/070d859c-d644-4b85-9350-e1ce6c37d1aa-kube-api-access-vmmbb\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:28.554903 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:28.554866 5727 generic.go:356] "Generic (PLEG): container finished" podID="070d859c-d644-4b85-9350-e1ce6c37d1aa" containerID="f904352cb3979a913b67cc7afb4ad5bba855c6bf8380d6d03723c5c750c31624" exitCode=0 Nov 02 23:01:28.555091 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:28.554913 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5dd28258-8478955cbc-s5k87" Nov 02 23:01:28.555091 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:28.554959 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5dd28258-8478955cbc-s5k87" event={"ID":"070d859c-d644-4b85-9350-e1ce6c37d1aa","Type":"ContainerDied","Data":"f904352cb3979a913b67cc7afb4ad5bba855c6bf8380d6d03723c5c750c31624"} Nov 02 23:01:28.555091 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:28.555019 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5dd28258-8478955cbc-s5k87" event={"ID":"070d859c-d644-4b85-9350-e1ce6c37d1aa","Type":"ContainerDied","Data":"e370faea54404192ea0ff1abba7c16dfd14e9c5c8e6b60c99af3bdf73c1a2b11"} Nov 02 23:01:28.555091 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:28.555048 5727 scope.go:122] "RemoveContainer" containerID="f904352cb3979a913b67cc7afb4ad5bba855c6bf8380d6d03723c5c750c31624" Nov 02 23:01:28.567850 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:28.567830 5727 scope.go:122] "RemoveContainer" containerID="f904352cb3979a913b67cc7afb4ad5bba855c6bf8380d6d03723c5c750c31624" Nov 02 23:01:28.568190 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:01:28.568169 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f904352cb3979a913b67cc7afb4ad5bba855c6bf8380d6d03723c5c750c31624\": container with ID starting with f904352cb3979a913b67cc7afb4ad5bba855c6bf8380d6d03723c5c750c31624 not found: ID does not exist" containerID="f904352cb3979a913b67cc7afb4ad5bba855c6bf8380d6d03723c5c750c31624" Nov 02 23:01:28.568253 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:28.568206 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f904352cb3979a913b67cc7afb4ad5bba855c6bf8380d6d03723c5c750c31624"} err="failed to get container status \"f904352cb3979a913b67cc7afb4ad5bba855c6bf8380d6d03723c5c750c31624\": rpc error: code = NotFound desc = could not find container \"f904352cb3979a913b67cc7afb4ad5bba855c6bf8380d6d03723c5c750c31624\": container with ID starting with f904352cb3979a913b67cc7afb4ad5bba855c6bf8380d6d03723c5c750c31624 not found: ID does not exist" Nov 02 23:01:28.719114 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:28.719024 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5dd28258-8478955cbc-s5k87"] Nov 02 23:01:28.723878 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:28.723853 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5dd28258-8478955cbc-s5k87"] Nov 02 23:01:29.882010 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:29.881975 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="070d859c-d644-4b85-9350-e1ce6c37d1aa" path="/var/lib/kubelet/pods/070d859c-d644-4b85-9350-e1ce6c37d1aa/volumes" Nov 02 23:01:30.307080 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:30.306994 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-fed9e8ee-787876d555-j2dhc" podUID="b32b75be-1f34-4ca0-8494-dd6bf7f31524" containerName="payload-processing" containerID="cri-o://5ecaa6b9b649f25d33156e6ed21c8bf36bf579a682dbd4292c4ec2af165acf8c" gracePeriod=25 Nov 02 23:01:30.412919 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:30.412878 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed9e8ee-644dc48579-kvq2w" podUID="443ff9dc-8fc9-447a-b0e4-1606cd671b61" containerName="payload-pre-processing" containerID="cri-o://b3b2953d92b8da84617660f6476e3e7865af491654d8a24520f741b79b797e68" gracePeriod=25 Nov 02 23:01:30.576504 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:30.576481 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fed9e8ee-787876d555-j2dhc" Nov 02 23:01:30.580295 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:30.580269 5727 generic.go:356] "Generic (PLEG): container finished" podID="443ff9dc-8fc9-447a-b0e4-1606cd671b61" containerID="b3b2953d92b8da84617660f6476e3e7865af491654d8a24520f741b79b797e68" exitCode=0 Nov 02 23:01:30.580416 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:30.580340 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed9e8ee-644dc48579-kvq2w" event={"ID":"443ff9dc-8fc9-447a-b0e4-1606cd671b61","Type":"ContainerDied","Data":"b3b2953d92b8da84617660f6476e3e7865af491654d8a24520f741b79b797e68"} Nov 02 23:01:30.581881 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:30.581844 5727 generic.go:356] "Generic (PLEG): container finished" podID="b32b75be-1f34-4ca0-8494-dd6bf7f31524" containerID="5ecaa6b9b649f25d33156e6ed21c8bf36bf579a682dbd4292c4ec2af165acf8c" exitCode=0 Nov 02 23:01:30.581881 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:30.581867 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fed9e8ee-787876d555-j2dhc" event={"ID":"b32b75be-1f34-4ca0-8494-dd6bf7f31524","Type":"ContainerDied","Data":"5ecaa6b9b649f25d33156e6ed21c8bf36bf579a682dbd4292c4ec2af165acf8c"} Nov 02 23:01:30.582050 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:30.581852 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fed9e8ee-787876d555-j2dhc" Nov 02 23:01:30.582050 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:30.581913 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fed9e8ee-787876d555-j2dhc" event={"ID":"b32b75be-1f34-4ca0-8494-dd6bf7f31524","Type":"ContainerDied","Data":"8950c35029ab561374e77641f70f2753582857ad2d8fa7318a553756a0e65622"} Nov 02 23:01:30.582050 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:30.581956 5727 scope.go:122] "RemoveContainer" containerID="5ecaa6b9b649f25d33156e6ed21c8bf36bf579a682dbd4292c4ec2af165acf8c" Nov 02 23:01:30.593318 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:30.593293 5727 scope.go:122] "RemoveContainer" containerID="5ecaa6b9b649f25d33156e6ed21c8bf36bf579a682dbd4292c4ec2af165acf8c" Nov 02 23:01:30.593880 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:01:30.593672 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ecaa6b9b649f25d33156e6ed21c8bf36bf579a682dbd4292c4ec2af165acf8c\": container with ID starting with 5ecaa6b9b649f25d33156e6ed21c8bf36bf579a682dbd4292c4ec2af165acf8c not found: ID does not exist" containerID="5ecaa6b9b649f25d33156e6ed21c8bf36bf579a682dbd4292c4ec2af165acf8c" Nov 02 23:01:30.593880 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:30.593712 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ecaa6b9b649f25d33156e6ed21c8bf36bf579a682dbd4292c4ec2af165acf8c"} err="failed to get container status \"5ecaa6b9b649f25d33156e6ed21c8bf36bf579a682dbd4292c4ec2af165acf8c\": rpc error: code = NotFound desc = could not find container \"5ecaa6b9b649f25d33156e6ed21c8bf36bf579a682dbd4292c4ec2af165acf8c\": container with ID starting with 5ecaa6b9b649f25d33156e6ed21c8bf36bf579a682dbd4292c4ec2af165acf8c not found: ID does not exist" Nov 02 23:01:30.661385 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:30.661354 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b32b75be-1f34-4ca0-8494-dd6bf7f31524-kube-api-access-pr2fh\" (UniqueName: \"kubernetes.io/projected/b32b75be-1f34-4ca0-8494-dd6bf7f31524-kube-api-access-pr2fh\") pod \"b32b75be-1f34-4ca0-8494-dd6bf7f31524\" (UID: \"b32b75be-1f34-4ca0-8494-dd6bf7f31524\") " Nov 02 23:01:30.661550 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:30.661422 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b32b75be-1f34-4ca0-8494-dd6bf7f31524-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b32b75be-1f34-4ca0-8494-dd6bf7f31524-plugins-config-volume\") pod \"b32b75be-1f34-4ca0-8494-dd6bf7f31524\" (UID: \"b32b75be-1f34-4ca0-8494-dd6bf7f31524\") " Nov 02 23:01:30.661811 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:30.661784 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b32b75be-1f34-4ca0-8494-dd6bf7f31524-plugins-config-volume" pod "b32b75be-1f34-4ca0-8494-dd6bf7f31524" (UID: "b32b75be-1f34-4ca0-8494-dd6bf7f31524"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:01:30.663859 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:30.663834 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b32b75be-1f34-4ca0-8494-dd6bf7f31524-kube-api-access-pr2fh" pod "b32b75be-1f34-4ca0-8494-dd6bf7f31524" (UID: "b32b75be-1f34-4ca0-8494-dd6bf7f31524"). InnerVolumeSpecName "kube-api-access-pr2fh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:01:30.684794 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:30.684776 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed9e8ee-644dc48579-kvq2w" Nov 02 23:01:30.762320 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:30.762292 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/443ff9dc-8fc9-447a-b0e4-1606cd671b61-kube-api-access-b7ccf\" (UniqueName: \"kubernetes.io/projected/443ff9dc-8fc9-447a-b0e4-1606cd671b61-kube-api-access-b7ccf\") pod \"443ff9dc-8fc9-447a-b0e4-1606cd671b61\" (UID: \"443ff9dc-8fc9-447a-b0e4-1606cd671b61\") " Nov 02 23:01:30.762516 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:30.762417 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/443ff9dc-8fc9-447a-b0e4-1606cd671b61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/443ff9dc-8fc9-447a-b0e4-1606cd671b61-plugins-config-volume\") pod \"443ff9dc-8fc9-447a-b0e4-1606cd671b61\" (UID: \"443ff9dc-8fc9-447a-b0e4-1606cd671b61\") " Nov 02 23:01:30.762773 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:30.762751 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr2fh\" (UniqueName: \"kubernetes.io/projected/b32b75be-1f34-4ca0-8494-dd6bf7f31524-kube-api-access-pr2fh\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:30.762773 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:30.762773 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b32b75be-1f34-4ca0-8494-dd6bf7f31524-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:30.762920 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:30.762817 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/443ff9dc-8fc9-447a-b0e4-1606cd671b61-plugins-config-volume" pod "443ff9dc-8fc9-447a-b0e4-1606cd671b61" (UID: "443ff9dc-8fc9-447a-b0e4-1606cd671b61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:01:30.765782 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:30.765753 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/443ff9dc-8fc9-447a-b0e4-1606cd671b61-kube-api-access-b7ccf" pod "443ff9dc-8fc9-447a-b0e4-1606cd671b61" (UID: "443ff9dc-8fc9-447a-b0e4-1606cd671b61"). InnerVolumeSpecName "kube-api-access-b7ccf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:01:30.863947 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:30.863865 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7ccf\" (UniqueName: \"kubernetes.io/projected/443ff9dc-8fc9-447a-b0e4-1606cd671b61-kube-api-access-b7ccf\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:30.863947 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:30.863892 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/443ff9dc-8fc9-447a-b0e4-1606cd671b61-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:31.030489 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:31.030452 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fed9e8ee-787876d555-j2dhc"] Nov 02 23:01:31.034928 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:31.034903 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fed9e8ee-787876d555-j2dhc"] Nov 02 23:01:31.591200 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:31.591108 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed9e8ee-644dc48579-kvq2w" event={"ID":"443ff9dc-8fc9-447a-b0e4-1606cd671b61","Type":"ContainerDied","Data":"77fc931cecafd6ebd12c6ff62d194a7eeede36b0d92bc02543bd74334a3f7a23"} Nov 02 23:01:31.591200 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:31.591180 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed9e8ee-644dc48579-kvq2w" Nov 02 23:01:31.591404 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:31.591182 5727 scope.go:122] "RemoveContainer" containerID="b3b2953d92b8da84617660f6476e3e7865af491654d8a24520f741b79b797e68" Nov 02 23:01:31.756298 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:31.756267 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fed9e8ee-644dc48579-kvq2w"] Nov 02 23:01:31.760447 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:31.760421 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fed9e8ee-644dc48579-kvq2w"] Nov 02 23:01:31.881878 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:31.881810 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="443ff9dc-8fc9-447a-b0e4-1606cd671b61" path="/var/lib/kubelet/pods/443ff9dc-8fc9-447a-b0e4-1606cd671b61/volumes" Nov 02 23:01:31.882197 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:31.882183 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b32b75be-1f34-4ca0-8494-dd6bf7f31524" path="/var/lib/kubelet/pods/b32b75be-1f34-4ca0-8494-dd6bf7f31524/volumes" Nov 02 23:01:35.171432 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:35.171398 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97"] Nov 02 23:01:35.171973 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:35.171696 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97" podUID="3fa69d14-76a5-471d-b452-bbbacf892687" containerName="maas-api" containerID="cri-o://0ceeda311ed8dac240dfc22d5dfd25408c9583139d977e31d2cd6adcc39ad0cd" gracePeriod=30 Nov 02 23:01:35.219883 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:35.219842 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f779f4ae-5775cbf6d9-6lzz5"] Nov 02 23:01:35.304529 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:35.304497 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f779f4ae-67fccdfcc9-gt7n9"] Nov 02 23:01:35.445661 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:35.445631 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97" Nov 02 23:01:35.628270 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:35.628239 5727 generic.go:356] "Generic (PLEG): container finished" podID="3fa69d14-76a5-471d-b452-bbbacf892687" containerID="0ceeda311ed8dac240dfc22d5dfd25408c9583139d977e31d2cd6adcc39ad0cd" exitCode=0 Nov 02 23:01:35.628422 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:35.628290 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97" Nov 02 23:01:35.628422 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:35.628324 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97" event={"ID":"3fa69d14-76a5-471d-b452-bbbacf892687","Type":"ContainerDied","Data":"0ceeda311ed8dac240dfc22d5dfd25408c9583139d977e31d2cd6adcc39ad0cd"} Nov 02 23:01:35.628422 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:35.628382 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97" event={"ID":"3fa69d14-76a5-471d-b452-bbbacf892687","Type":"ContainerDied","Data":"589986e9e493a790aea58d30efa5854c4d0d668e6dcddd00c39d7bddd32bce76"} Nov 02 23:01:35.628422 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:35.628410 5727 scope.go:122] "RemoveContainer" containerID="0ceeda311ed8dac240dfc22d5dfd25408c9583139d977e31d2cd6adcc39ad0cd" Nov 02 23:01:35.630699 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:35.630675 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fa69d14-76a5-471d-b452-bbbacf892687-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa69d14-76a5-471d-b452-bbbacf892687-openshift-service-ca\") pod \"3fa69d14-76a5-471d-b452-bbbacf892687\" (UID: \"3fa69d14-76a5-471d-b452-bbbacf892687\") " Nov 02 23:01:35.630815 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:35.630733 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fa69d14-76a5-471d-b452-bbbacf892687-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fa69d14-76a5-471d-b452-bbbacf892687-maas-api-tls\") pod \"3fa69d14-76a5-471d-b452-bbbacf892687\" (UID: \"3fa69d14-76a5-471d-b452-bbbacf892687\") " Nov 02 23:01:35.630870 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:35.630843 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa69d14-76a5-471d-b452-bbbacf892687-kube-api-access-d4scg\" (UniqueName: \"kubernetes.io/projected/3fa69d14-76a5-471d-b452-bbbacf892687-kube-api-access-d4scg\") pod \"3fa69d14-76a5-471d-b452-bbbacf892687\" (UID: \"3fa69d14-76a5-471d-b452-bbbacf892687\") " Nov 02 23:01:35.630954 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:35.630934 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fa69d14-76a5-471d-b452-bbbacf892687-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fa69d14-76a5-471d-b452-bbbacf892687-metrics-certs\") pod \"3fa69d14-76a5-471d-b452-bbbacf892687\" (UID: \"3fa69d14-76a5-471d-b452-bbbacf892687\") " Nov 02 23:01:35.631101 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:35.631070 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa69d14-76a5-471d-b452-bbbacf892687-openshift-service-ca" pod "3fa69d14-76a5-471d-b452-bbbacf892687" (UID: "3fa69d14-76a5-471d-b452-bbbacf892687"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:01:35.631438 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:35.631372 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa69d14-76a5-471d-b452-bbbacf892687-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:35.632919 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:35.632897 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa69d14-76a5-471d-b452-bbbacf892687-kube-api-access-d4scg" pod "3fa69d14-76a5-471d-b452-bbbacf892687" (UID: "3fa69d14-76a5-471d-b452-bbbacf892687"). InnerVolumeSpecName "kube-api-access-d4scg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:01:35.633041 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:35.633025 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fa69d14-76a5-471d-b452-bbbacf892687-maas-api-tls" pod "3fa69d14-76a5-471d-b452-bbbacf892687" (UID: "3fa69d14-76a5-471d-b452-bbbacf892687"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:01:35.633535 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:35.633505 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fa69d14-76a5-471d-b452-bbbacf892687-metrics-certs" pod "3fa69d14-76a5-471d-b452-bbbacf892687" (UID: "3fa69d14-76a5-471d-b452-bbbacf892687"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:01:35.657667 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:35.657640 5727 scope.go:122] "RemoveContainer" containerID="0ceeda311ed8dac240dfc22d5dfd25408c9583139d977e31d2cd6adcc39ad0cd" Nov 02 23:01:35.658026 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:01:35.657996 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ceeda311ed8dac240dfc22d5dfd25408c9583139d977e31d2cd6adcc39ad0cd\": container with ID starting with 0ceeda311ed8dac240dfc22d5dfd25408c9583139d977e31d2cd6adcc39ad0cd not found: ID does not exist" containerID="0ceeda311ed8dac240dfc22d5dfd25408c9583139d977e31d2cd6adcc39ad0cd" Nov 02 23:01:35.658116 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:35.658033 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ceeda311ed8dac240dfc22d5dfd25408c9583139d977e31d2cd6adcc39ad0cd"} err="failed to get container status \"0ceeda311ed8dac240dfc22d5dfd25408c9583139d977e31d2cd6adcc39ad0cd\": rpc error: code = NotFound desc = could not find container \"0ceeda311ed8dac240dfc22d5dfd25408c9583139d977e31d2cd6adcc39ad0cd\": container with ID starting with 0ceeda311ed8dac240dfc22d5dfd25408c9583139d977e31d2cd6adcc39ad0cd not found: ID does not exist" Nov 02 23:01:35.732354 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:35.732272 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4scg\" (UniqueName: \"kubernetes.io/projected/3fa69d14-76a5-471d-b452-bbbacf892687-kube-api-access-d4scg\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:35.732354 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:35.732301 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fa69d14-76a5-471d-b452-bbbacf892687-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:35.732354 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:35.732311 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fa69d14-76a5-471d-b452-bbbacf892687-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:36.048214 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:36.048137 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97"] Nov 02 23:01:36.053072 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:36.053050 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f779f4ae-6cfbfd8c6b-22p97"] Nov 02 23:01:37.883474 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:37.883435 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fa69d14-76a5-471d-b452-bbbacf892687" path="/var/lib/kubelet/pods/3fa69d14-76a5-471d-b452-bbbacf892687/volumes" Nov 02 23:01:39.185084 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.185049 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-08c2c6-7bcc7fc5dc-59fbq"] Nov 02 23:01:39.185543 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.185530 5727 cpu_manager.go:414] "containerMap: removing container" podUID="0b073ad5-b50e-4b8a-a750-9cbc29638e11" containerName="payload-pre-processing" Nov 02 23:01:39.185587 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.185544 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b073ad5-b50e-4b8a-a750-9cbc29638e11" containerName="payload-pre-processing" Nov 02 23:01:39.185587 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.185568 5727 cpu_manager.go:414] "containerMap: removing container" podUID="1241a260-7110-40dc-9f4b-99057b083a0e" containerName="payload-processing" Nov 02 23:01:39.185587 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.185573 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1241a260-7110-40dc-9f4b-99057b083a0e" containerName="payload-processing" Nov 02 23:01:39.185682 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.185590 5727 cpu_manager.go:414] "containerMap: removing container" podUID="3fa69d14-76a5-471d-b452-bbbacf892687" containerName="maas-api" Nov 02 23:01:39.185682 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.185595 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa69d14-76a5-471d-b452-bbbacf892687" containerName="maas-api" Nov 02 23:01:39.185682 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.185606 5727 cpu_manager.go:414] "containerMap: removing container" podUID="b32b75be-1f34-4ca0-8494-dd6bf7f31524" containerName="payload-processing" Nov 02 23:01:39.185682 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.185612 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b32b75be-1f34-4ca0-8494-dd6bf7f31524" containerName="payload-processing" Nov 02 23:01:39.185682 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.185620 5727 cpu_manager.go:414] "containerMap: removing container" podUID="070d859c-d644-4b85-9350-e1ce6c37d1aa" containerName="payload-pre-processing" Nov 02 23:01:39.185682 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.185624 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="070d859c-d644-4b85-9350-e1ce6c37d1aa" containerName="payload-pre-processing" Nov 02 23:01:39.185682 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.185633 5727 cpu_manager.go:414] "containerMap: removing container" podUID="2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c" containerName="maas-api" Nov 02 23:01:39.185682 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.185637 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c" containerName="maas-api" Nov 02 23:01:39.185682 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.185647 5727 cpu_manager.go:414] "containerMap: removing container" podUID="443ff9dc-8fc9-447a-b0e4-1606cd671b61" containerName="payload-pre-processing" Nov 02 23:01:39.185682 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.185651 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="443ff9dc-8fc9-447a-b0e4-1606cd671b61" containerName="payload-pre-processing" Nov 02 23:01:39.185963 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.185714 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa69d14-76a5-471d-b452-bbbacf892687" containerName="maas-api" Nov 02 23:01:39.185963 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.185722 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="b32b75be-1f34-4ca0-8494-dd6bf7f31524" containerName="payload-processing" Nov 02 23:01:39.185963 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.185728 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="070d859c-d644-4b85-9350-e1ce6c37d1aa" containerName="payload-pre-processing" Nov 02 23:01:39.185963 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.185735 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="2644cc75-0d2f-4a15-b7d9-c9faa7d57d1c" containerName="maas-api" Nov 02 23:01:39.185963 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.185742 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="443ff9dc-8fc9-447a-b0e4-1606cd671b61" containerName="payload-pre-processing" Nov 02 23:01:39.185963 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.185747 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b073ad5-b50e-4b8a-a750-9cbc29638e11" containerName="payload-pre-processing" Nov 02 23:01:39.185963 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.185754 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="1241a260-7110-40dc-9f4b-99057b083a0e" containerName="payload-processing" Nov 02 23:01:39.190483 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.190466 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-08c2c6-7bcc7fc5dc-59fbq" Nov 02 23:01:39.242994 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.242964 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-08c2c6\"" Nov 02 23:01:39.246437 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.246413 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-08c2c6-7bcc7fc5dc-59fbq"] Nov 02 23:01:39.268071 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.268030 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j74vs\" (UniqueName: \"kubernetes.io/projected/73abdc63-f4c0-48b5-913b-136f46987d0e-kube-api-access-j74vs\") pod \"payload-pre-processing-e2e-worker-w2-08c2c6-7bcc7fc5dc-59fbq\" (UID: \"73abdc63-f4c0-48b5-913b-136f46987d0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-08c2c6-7bcc7fc5dc-59fbq" Nov 02 23:01:39.268287 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.268269 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73abdc63-f4c0-48b5-913b-136f46987d0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-08c2c6-7bcc7fc5dc-59fbq\" (UID: \"73abdc63-f4c0-48b5-913b-136f46987d0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-08c2c6-7bcc7fc5dc-59fbq" Nov 02 23:01:39.292655 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.292627 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-08c2c6-dockercfg-ls49k\"" Nov 02 23:01:39.331014 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.330984 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-08c2c6-7b8bd45695-rf87d"] Nov 02 23:01:39.334656 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.334641 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-08c2c6-7b8bd45695-rf87d" Nov 02 23:01:39.369712 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.369681 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mq8m\" (UniqueName: \"kubernetes.io/projected/2158b9be-b6e4-4a86-8c9b-441abe53d862-kube-api-access-2mq8m\") pod \"payload-processing-e2e-worker-w2-08c2c6-7b8bd45695-rf87d\" (UID: \"2158b9be-b6e4-4a86-8c9b-441abe53d862\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-08c2c6-7b8bd45695-rf87d" Nov 02 23:01:39.369898 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.369718 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73abdc63-f4c0-48b5-913b-136f46987d0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-08c2c6-7bcc7fc5dc-59fbq\" (UID: \"73abdc63-f4c0-48b5-913b-136f46987d0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-08c2c6-7bcc7fc5dc-59fbq" Nov 02 23:01:39.369898 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.369750 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j74vs\" (UniqueName: \"kubernetes.io/projected/73abdc63-f4c0-48b5-913b-136f46987d0e-kube-api-access-j74vs\") pod \"payload-pre-processing-e2e-worker-w2-08c2c6-7bcc7fc5dc-59fbq\" (UID: \"73abdc63-f4c0-48b5-913b-136f46987d0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-08c2c6-7bcc7fc5dc-59fbq" Nov 02 23:01:39.370016 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.369940 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2158b9be-b6e4-4a86-8c9b-441abe53d862-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-08c2c6-7b8bd45695-rf87d\" (UID: \"2158b9be-b6e4-4a86-8c9b-441abe53d862\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-08c2c6-7b8bd45695-rf87d" Nov 02 23:01:39.370455 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.370439 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73abdc63-f4c0-48b5-913b-136f46987d0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-08c2c6-7bcc7fc5dc-59fbq\" (UID: \"73abdc63-f4c0-48b5-913b-136f46987d0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-08c2c6-7bcc7fc5dc-59fbq" Nov 02 23:01:39.392893 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.392861 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-08c2c6-7b8bd45695-rf87d"] Nov 02 23:01:39.431045 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.431017 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j74vs\" (UniqueName: \"kubernetes.io/projected/73abdc63-f4c0-48b5-913b-136f46987d0e-kube-api-access-j74vs\") pod \"payload-pre-processing-e2e-worker-w2-08c2c6-7bcc7fc5dc-59fbq\" (UID: \"73abdc63-f4c0-48b5-913b-136f46987d0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-08c2c6-7bcc7fc5dc-59fbq" Nov 02 23:01:39.471305 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.471206 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2158b9be-b6e4-4a86-8c9b-441abe53d862-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-08c2c6-7b8bd45695-rf87d\" (UID: \"2158b9be-b6e4-4a86-8c9b-441abe53d862\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-08c2c6-7b8bd45695-rf87d" Nov 02 23:01:39.471305 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.471279 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mq8m\" (UniqueName: \"kubernetes.io/projected/2158b9be-b6e4-4a86-8c9b-441abe53d862-kube-api-access-2mq8m\") pod \"payload-processing-e2e-worker-w2-08c2c6-7b8bd45695-rf87d\" (UID: \"2158b9be-b6e4-4a86-8c9b-441abe53d862\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-08c2c6-7b8bd45695-rf87d" Nov 02 23:01:39.471883 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.471862 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2158b9be-b6e4-4a86-8c9b-441abe53d862-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-08c2c6-7b8bd45695-rf87d\" (UID: \"2158b9be-b6e4-4a86-8c9b-441abe53d862\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-08c2c6-7b8bd45695-rf87d" Nov 02 23:01:39.504358 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.504325 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-08c2c6-7bcc7fc5dc-59fbq" Nov 02 23:01:39.532198 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.532166 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mq8m\" (UniqueName: \"kubernetes.io/projected/2158b9be-b6e4-4a86-8c9b-441abe53d862-kube-api-access-2mq8m\") pod \"payload-processing-e2e-worker-w2-08c2c6-7b8bd45695-rf87d\" (UID: \"2158b9be-b6e4-4a86-8c9b-441abe53d862\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-08c2c6-7b8bd45695-rf87d" Nov 02 23:01:39.655657 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.655623 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-08c2c6-7b8bd45695-rf87d" Nov 02 23:01:39.883692 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:39.883668 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-08c2c6-7bcc7fc5dc-59fbq"] Nov 02 23:01:39.885548 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:01:39.885514 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73abdc63_f4c0_48b5_913b_136f46987d0e.slice/crio-505df05fbb73d87235ae4ef31e2130ead7ad99e1a1ee218875b8bf79826d8e0d WatchSource:0}: Error finding container 505df05fbb73d87235ae4ef31e2130ead7ad99e1a1ee218875b8bf79826d8e0d: Status 404 returned error can't find the container with id 505df05fbb73d87235ae4ef31e2130ead7ad99e1a1ee218875b8bf79826d8e0d Nov 02 23:01:40.034448 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.034426 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-08c2c6-7b8bd45695-rf87d"] Nov 02 23:01:40.037158 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:01:40.037114 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2158b9be_b6e4_4a86_8c9b_441abe53d862.slice/crio-86000be111db86b600923364726ae722787f681e3dd439b024c970fbf9f27064 WatchSource:0}: Error finding container 86000be111db86b600923364726ae722787f681e3dd439b024c970fbf9f27064: Status 404 returned error can't find the container with id 86000be111db86b600923364726ae722787f681e3dd439b024c970fbf9f27064 Nov 02 23:01:40.220723 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.220638 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f779f4ae-5775cbf6d9-6lzz5" podUID="6fe11484-ba7f-4b38-8a9c-1bf3893578a5" containerName="payload-processing" containerID="cri-o://715f6260402a9f6ae62646afe75f9ef733e202ee8b3cb06f9d0f8fec546132ff" gracePeriod=25 Nov 02 23:01:40.305589 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.305533 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f779f4ae-67fccdfcc9-gt7n9" podUID="3ea4766f-c234-4032-bd30-36e9e9a9896a" containerName="payload-pre-processing" containerID="cri-o://587ceadd0b84b0d9b32e3c62bd00cbe85d194ceffdc756359914db3dd86055d0" gracePeriod=25 Nov 02 23:01:40.595729 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.595698 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f779f4ae-5775cbf6d9-6lzz5" Nov 02 23:01:40.642698 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.642669 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f779f4ae-67fccdfcc9-gt7n9" Nov 02 23:01:40.664207 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.664167 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-08c2c6-7b8bd45695-rf87d" event={"ID":"2158b9be-b6e4-4a86-8c9b-441abe53d862","Type":"ContainerStarted","Data":"a508af4eb526befd1e233e6e87cfb358a6cbb646bac02982a0141e8b976733e4"} Nov 02 23:01:40.664400 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.664264 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-08c2c6-7b8bd45695-rf87d" event={"ID":"2158b9be-b6e4-4a86-8c9b-441abe53d862","Type":"ContainerStarted","Data":"86000be111db86b600923364726ae722787f681e3dd439b024c970fbf9f27064"} Nov 02 23:01:40.664400 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.664291 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-08c2c6-7b8bd45695-rf87d" Nov 02 23:01:40.665785 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.665754 5727 generic.go:356] "Generic (PLEG): container finished" podID="6fe11484-ba7f-4b38-8a9c-1bf3893578a5" containerID="715f6260402a9f6ae62646afe75f9ef733e202ee8b3cb06f9d0f8fec546132ff" exitCode=0 Nov 02 23:01:40.665908 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.665818 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f779f4ae-5775cbf6d9-6lzz5" Nov 02 23:01:40.665908 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.665827 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f779f4ae-5775cbf6d9-6lzz5" event={"ID":"6fe11484-ba7f-4b38-8a9c-1bf3893578a5","Type":"ContainerDied","Data":"715f6260402a9f6ae62646afe75f9ef733e202ee8b3cb06f9d0f8fec546132ff"} Nov 02 23:01:40.665908 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.665871 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f779f4ae-5775cbf6d9-6lzz5" event={"ID":"6fe11484-ba7f-4b38-8a9c-1bf3893578a5","Type":"ContainerDied","Data":"c24f0b5f8d4e9f46b675e71a58fb2869006df55b1bba93d66add4c1cb2b36fb7"} Nov 02 23:01:40.665908 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.665901 5727 scope.go:122] "RemoveContainer" containerID="715f6260402a9f6ae62646afe75f9ef733e202ee8b3cb06f9d0f8fec546132ff" Nov 02 23:01:40.667569 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.667548 5727 generic.go:356] "Generic (PLEG): container finished" podID="3ea4766f-c234-4032-bd30-36e9e9a9896a" containerID="587ceadd0b84b0d9b32e3c62bd00cbe85d194ceffdc756359914db3dd86055d0" exitCode=0 Nov 02 23:01:40.667680 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.667612 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f779f4ae-67fccdfcc9-gt7n9" Nov 02 23:01:40.667680 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.667635 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f779f4ae-67fccdfcc9-gt7n9" event={"ID":"3ea4766f-c234-4032-bd30-36e9e9a9896a","Type":"ContainerDied","Data":"587ceadd0b84b0d9b32e3c62bd00cbe85d194ceffdc756359914db3dd86055d0"} Nov 02 23:01:40.667784 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.667705 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f779f4ae-67fccdfcc9-gt7n9" event={"ID":"3ea4766f-c234-4032-bd30-36e9e9a9896a","Type":"ContainerDied","Data":"050220df6573ab244b5a9e84e431984f44dfa0991fb2e6136a73862cba7ca4a5"} Nov 02 23:01:40.669709 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.669684 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-08c2c6-7bcc7fc5dc-59fbq" event={"ID":"73abdc63-f4c0-48b5-913b-136f46987d0e","Type":"ContainerStarted","Data":"7cc0ea899a8f3af38ff939dd7c7a510e441b494d15102d09c114d803e9aae1b6"} Nov 02 23:01:40.669898 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.669870 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-08c2c6-7bcc7fc5dc-59fbq" event={"ID":"73abdc63-f4c0-48b5-913b-136f46987d0e","Type":"ContainerStarted","Data":"505df05fbb73d87235ae4ef31e2130ead7ad99e1a1ee218875b8bf79826d8e0d"} Nov 02 23:01:40.670021 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.669914 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-08c2c6-7bcc7fc5dc-59fbq" Nov 02 23:01:40.677825 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.677804 5727 scope.go:122] "RemoveContainer" containerID="715f6260402a9f6ae62646afe75f9ef733e202ee8b3cb06f9d0f8fec546132ff" Nov 02 23:01:40.678192 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:01:40.678170 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"715f6260402a9f6ae62646afe75f9ef733e202ee8b3cb06f9d0f8fec546132ff\": container with ID starting with 715f6260402a9f6ae62646afe75f9ef733e202ee8b3cb06f9d0f8fec546132ff not found: ID does not exist" containerID="715f6260402a9f6ae62646afe75f9ef733e202ee8b3cb06f9d0f8fec546132ff" Nov 02 23:01:40.678290 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.678201 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"715f6260402a9f6ae62646afe75f9ef733e202ee8b3cb06f9d0f8fec546132ff"} err="failed to get container status \"715f6260402a9f6ae62646afe75f9ef733e202ee8b3cb06f9d0f8fec546132ff\": rpc error: code = NotFound desc = could not find container \"715f6260402a9f6ae62646afe75f9ef733e202ee8b3cb06f9d0f8fec546132ff\": container with ID starting with 715f6260402a9f6ae62646afe75f9ef733e202ee8b3cb06f9d0f8fec546132ff not found: ID does not exist" Nov 02 23:01:40.678290 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.678224 5727 scope.go:122] "RemoveContainer" containerID="587ceadd0b84b0d9b32e3c62bd00cbe85d194ceffdc756359914db3dd86055d0" Nov 02 23:01:40.686752 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.686723 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fe11484-ba7f-4b38-8a9c-1bf3893578a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe11484-ba7f-4b38-8a9c-1bf3893578a5-plugins-config-volume\") pod \"6fe11484-ba7f-4b38-8a9c-1bf3893578a5\" (UID: \"6fe11484-ba7f-4b38-8a9c-1bf3893578a5\") " Nov 02 23:01:40.686888 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.686871 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fe11484-ba7f-4b38-8a9c-1bf3893578a5-kube-api-access-hmn8x\" (UniqueName: \"kubernetes.io/projected/6fe11484-ba7f-4b38-8a9c-1bf3893578a5-kube-api-access-hmn8x\") pod \"6fe11484-ba7f-4b38-8a9c-1bf3893578a5\" (UID: \"6fe11484-ba7f-4b38-8a9c-1bf3893578a5\") " Nov 02 23:01:40.686940 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.686927 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ea4766f-c234-4032-bd30-36e9e9a9896a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ea4766f-c234-4032-bd30-36e9e9a9896a-plugins-config-volume\") pod \"3ea4766f-c234-4032-bd30-36e9e9a9896a\" (UID: \"3ea4766f-c234-4032-bd30-36e9e9a9896a\") " Nov 02 23:01:40.686980 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.686963 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ea4766f-c234-4032-bd30-36e9e9a9896a-kube-api-access-gsbbm\" (UniqueName: \"kubernetes.io/projected/3ea4766f-c234-4032-bd30-36e9e9a9896a-kube-api-access-gsbbm\") pod \"3ea4766f-c234-4032-bd30-36e9e9a9896a\" (UID: \"3ea4766f-c234-4032-bd30-36e9e9a9896a\") " Nov 02 23:01:40.687756 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.687121 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fe11484-ba7f-4b38-8a9c-1bf3893578a5-plugins-config-volume" pod "6fe11484-ba7f-4b38-8a9c-1bf3893578a5" (UID: "6fe11484-ba7f-4b38-8a9c-1bf3893578a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:01:40.687756 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.687441 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ea4766f-c234-4032-bd30-36e9e9a9896a-plugins-config-volume" pod "3ea4766f-c234-4032-bd30-36e9e9a9896a" (UID: "3ea4766f-c234-4032-bd30-36e9e9a9896a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:01:40.687756 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.687541 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe11484-ba7f-4b38-8a9c-1bf3893578a5-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:40.688899 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.688855 5727 scope.go:122] "RemoveContainer" containerID="587ceadd0b84b0d9b32e3c62bd00cbe85d194ceffdc756359914db3dd86055d0" Nov 02 23:01:40.689437 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:01:40.689411 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"587ceadd0b84b0d9b32e3c62bd00cbe85d194ceffdc756359914db3dd86055d0\": container with ID starting with 587ceadd0b84b0d9b32e3c62bd00cbe85d194ceffdc756359914db3dd86055d0 not found: ID does not exist" containerID="587ceadd0b84b0d9b32e3c62bd00cbe85d194ceffdc756359914db3dd86055d0" Nov 02 23:01:40.689503 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.689448 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"587ceadd0b84b0d9b32e3c62bd00cbe85d194ceffdc756359914db3dd86055d0"} err="failed to get container status \"587ceadd0b84b0d9b32e3c62bd00cbe85d194ceffdc756359914db3dd86055d0\": rpc error: code = NotFound desc = could not find container \"587ceadd0b84b0d9b32e3c62bd00cbe85d194ceffdc756359914db3dd86055d0\": container with ID starting with 587ceadd0b84b0d9b32e3c62bd00cbe85d194ceffdc756359914db3dd86055d0 not found: ID does not exist" Nov 02 23:01:40.690213 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.690188 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fe11484-ba7f-4b38-8a9c-1bf3893578a5-kube-api-access-hmn8x" pod "6fe11484-ba7f-4b38-8a9c-1bf3893578a5" (UID: "6fe11484-ba7f-4b38-8a9c-1bf3893578a5"). InnerVolumeSpecName "kube-api-access-hmn8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:01:40.690285 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.690230 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea4766f-c234-4032-bd30-36e9e9a9896a-kube-api-access-gsbbm" pod "3ea4766f-c234-4032-bd30-36e9e9a9896a" (UID: "3ea4766f-c234-4032-bd30-36e9e9a9896a"). InnerVolumeSpecName "kube-api-access-gsbbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:01:40.788692 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.788602 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmn8x\" (UniqueName: \"kubernetes.io/projected/6fe11484-ba7f-4b38-8a9c-1bf3893578a5-kube-api-access-hmn8x\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:40.788692 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.788636 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ea4766f-c234-4032-bd30-36e9e9a9896a-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:40.788692 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.788649 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsbbm\" (UniqueName: \"kubernetes.io/projected/3ea4766f-c234-4032-bd30-36e9e9a9896a-kube-api-access-gsbbm\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:40.941873 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:40.941814 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-08c2c6-7bcc7fc5dc-59fbq" podStartSLOduration=1.941799585 podStartE2EDuration="1.941799585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:01:40.940374569 +0000 UTC m=+787.178872251" watchObservedRunningTime="2026-11-02 23:01:40.941799585 +0000 UTC m=+787.180297248" Nov 02 23:01:41.109272 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:41.109218 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-08c2c6-7b8bd45695-rf87d" podStartSLOduration=2.109205417 podStartE2EDuration="2.109205417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:01:41.107961876 +0000 UTC m=+787.346459558" watchObservedRunningTime="2026-11-02 23:01:41.109205417 +0000 UTC m=+787.347703119" Nov 02 23:01:41.225112 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:41.225074 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f779f4ae-5775cbf6d9-6lzz5"] Nov 02 23:01:41.227365 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:41.227343 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f779f4ae-5775cbf6d9-6lzz5"] Nov 02 23:01:41.334427 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:41.334385 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f779f4ae-67fccdfcc9-gt7n9"] Nov 02 23:01:41.340689 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:41.340654 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f779f4ae-67fccdfcc9-gt7n9"] Nov 02 23:01:41.881992 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:41.881958 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ea4766f-c234-4032-bd30-36e9e9a9896a" path="/var/lib/kubelet/pods/3ea4766f-c234-4032-bd30-36e9e9a9896a/volumes" Nov 02 23:01:41.882374 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:41.882360 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fe11484-ba7f-4b38-8a9c-1bf3893578a5" path="/var/lib/kubelet/pods/6fe11484-ba7f-4b38-8a9c-1bf3893578a5/volumes" Nov 02 23:01:43.464513 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:43.464474 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ec2f9a-76969d66d7-6b8dx"] Nov 02 23:01:43.465232 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:43.465215 5727 cpu_manager.go:414] "containerMap: removing container" podUID="3ea4766f-c234-4032-bd30-36e9e9a9896a" containerName="payload-pre-processing" Nov 02 23:01:43.465308 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:43.465237 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ea4766f-c234-4032-bd30-36e9e9a9896a" containerName="payload-pre-processing" Nov 02 23:01:43.465308 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:43.465259 5727 cpu_manager.go:414] "containerMap: removing container" podUID="6fe11484-ba7f-4b38-8a9c-1bf3893578a5" containerName="payload-processing" Nov 02 23:01:43.465308 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:43.465267 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fe11484-ba7f-4b38-8a9c-1bf3893578a5" containerName="payload-processing" Nov 02 23:01:43.465407 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:43.465395 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fe11484-ba7f-4b38-8a9c-1bf3893578a5" containerName="payload-processing" Nov 02 23:01:43.465440 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:43.465419 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ea4766f-c234-4032-bd30-36e9e9a9896a" containerName="payload-pre-processing" Nov 02 23:01:43.470538 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:43.470516 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ec2f9a-76969d66d7-6b8dx" Nov 02 23:01:43.514586 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:43.514556 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cdb3d15-9771-4a60-8e05-66895477704a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ec2f9a-76969d66d7-6b8dx\" (UID: \"1cdb3d15-9771-4a60-8e05-66895477704a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ec2f9a-76969d66d7-6b8dx" Nov 02 23:01:43.514795 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:43.514628 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjjzh\" (UniqueName: \"kubernetes.io/projected/1cdb3d15-9771-4a60-8e05-66895477704a-kube-api-access-sjjzh\") pod \"payload-pre-processing-e2e-worker-w0-ec2f9a-76969d66d7-6b8dx\" (UID: \"1cdb3d15-9771-4a60-8e05-66895477704a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ec2f9a-76969d66d7-6b8dx" Nov 02 23:01:43.525113 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:43.525078 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ec2f9a-76969d66d7-6b8dx"] Nov 02 23:01:43.570940 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:43.570904 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-ec2f9a\"" Nov 02 23:01:43.571186 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:43.571052 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-ec2f9a-dockercfg-pmb5v\"" Nov 02 23:01:43.608448 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:43.608414 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ec2f9a-589bcfb455-2r9sr"] Nov 02 23:01:43.612105 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:43.612088 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ec2f9a-589bcfb455-2r9sr" Nov 02 23:01:43.615311 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:43.615284 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjjzh\" (UniqueName: \"kubernetes.io/projected/1cdb3d15-9771-4a60-8e05-66895477704a-kube-api-access-sjjzh\") pod \"payload-pre-processing-e2e-worker-w0-ec2f9a-76969d66d7-6b8dx\" (UID: \"1cdb3d15-9771-4a60-8e05-66895477704a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ec2f9a-76969d66d7-6b8dx" Nov 02 23:01:43.615543 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:43.615399 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cdb3d15-9771-4a60-8e05-66895477704a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ec2f9a-76969d66d7-6b8dx\" (UID: \"1cdb3d15-9771-4a60-8e05-66895477704a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ec2f9a-76969d66d7-6b8dx" Nov 02 23:01:43.616380 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:43.616356 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cdb3d15-9771-4a60-8e05-66895477704a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ec2f9a-76969d66d7-6b8dx\" (UID: \"1cdb3d15-9771-4a60-8e05-66895477704a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ec2f9a-76969d66d7-6b8dx" Nov 02 23:01:43.670443 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:43.670409 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ec2f9a-589bcfb455-2r9sr"] Nov 02 23:01:43.677421 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:43.677380 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjjzh\" (UniqueName: \"kubernetes.io/projected/1cdb3d15-9771-4a60-8e05-66895477704a-kube-api-access-sjjzh\") pod \"payload-pre-processing-e2e-worker-w0-ec2f9a-76969d66d7-6b8dx\" (UID: \"1cdb3d15-9771-4a60-8e05-66895477704a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ec2f9a-76969d66d7-6b8dx" Nov 02 23:01:43.717008 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:43.716913 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c483a352-ae64-4bf1-bc45-6950f022114d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ec2f9a-589bcfb455-2r9sr\" (UID: \"c483a352-ae64-4bf1-bc45-6950f022114d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ec2f9a-589bcfb455-2r9sr" Nov 02 23:01:43.717008 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:43.717002 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs6qg\" (UniqueName: \"kubernetes.io/projected/c483a352-ae64-4bf1-bc45-6950f022114d-kube-api-access-rs6qg\") pod \"payload-processing-e2e-worker-w0-ec2f9a-589bcfb455-2r9sr\" (UID: \"c483a352-ae64-4bf1-bc45-6950f022114d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ec2f9a-589bcfb455-2r9sr" Nov 02 23:01:43.784779 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:43.784745 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ec2f9a-76969d66d7-6b8dx" Nov 02 23:01:43.817797 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:43.817764 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c483a352-ae64-4bf1-bc45-6950f022114d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ec2f9a-589bcfb455-2r9sr\" (UID: \"c483a352-ae64-4bf1-bc45-6950f022114d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ec2f9a-589bcfb455-2r9sr" Nov 02 23:01:43.817948 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:43.817822 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs6qg\" (UniqueName: \"kubernetes.io/projected/c483a352-ae64-4bf1-bc45-6950f022114d-kube-api-access-rs6qg\") pod \"payload-processing-e2e-worker-w0-ec2f9a-589bcfb455-2r9sr\" (UID: \"c483a352-ae64-4bf1-bc45-6950f022114d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ec2f9a-589bcfb455-2r9sr" Nov 02 23:01:43.821373 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:43.819093 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c483a352-ae64-4bf1-bc45-6950f022114d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ec2f9a-589bcfb455-2r9sr\" (UID: \"c483a352-ae64-4bf1-bc45-6950f022114d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ec2f9a-589bcfb455-2r9sr" Nov 02 23:01:43.880477 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:43.880432 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs6qg\" (UniqueName: \"kubernetes.io/projected/c483a352-ae64-4bf1-bc45-6950f022114d-kube-api-access-rs6qg\") pod \"payload-processing-e2e-worker-w0-ec2f9a-589bcfb455-2r9sr\" (UID: \"c483a352-ae64-4bf1-bc45-6950f022114d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ec2f9a-589bcfb455-2r9sr" Nov 02 23:01:43.926655 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:43.926613 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ec2f9a-589bcfb455-2r9sr" Nov 02 23:01:44.164934 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:44.164880 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ec2f9a-76969d66d7-6b8dx"] Nov 02 23:01:44.166285 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:01:44.166260 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cdb3d15_9771_4a60_8e05_66895477704a.slice/crio-40dd4136b38074f14e472d81603a1c9ca73adce298851e8229769310477107b9 WatchSource:0}: Error finding container 40dd4136b38074f14e472d81603a1c9ca73adce298851e8229769310477107b9: Status 404 returned error can't find the container with id 40dd4136b38074f14e472d81603a1c9ca73adce298851e8229769310477107b9 Nov 02 23:01:44.303097 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:44.303076 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ec2f9a-589bcfb455-2r9sr"] Nov 02 23:01:44.701504 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:44.701470 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ec2f9a-589bcfb455-2r9sr" event={"ID":"c483a352-ae64-4bf1-bc45-6950f022114d","Type":"ContainerStarted","Data":"c1787a0827f8e581b953ddd49a631dee38bb09ecd2ff03529dee301cbb3a500c"} Nov 02 23:01:44.702005 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:44.701539 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-ec2f9a-589bcfb455-2r9sr" Nov 02 23:01:44.702005 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:44.701565 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ec2f9a-589bcfb455-2r9sr" event={"ID":"c483a352-ae64-4bf1-bc45-6950f022114d","Type":"ContainerStarted","Data":"0e3672d8b8e278d5eafb72e7e5b14b827331265843fd30d66ac29ee21c9a2af4"} Nov 02 23:01:44.703278 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:44.703253 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ec2f9a-76969d66d7-6b8dx" event={"ID":"1cdb3d15-9771-4a60-8e05-66895477704a","Type":"ContainerStarted","Data":"66fe2597719c466391725b0c5c7442dca469ecae625c17e5969c73a8c558ba3b"} Nov 02 23:01:44.703396 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:44.703306 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ec2f9a-76969d66d7-6b8dx" Nov 02 23:01:44.703396 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:44.703330 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ec2f9a-76969d66d7-6b8dx" event={"ID":"1cdb3d15-9771-4a60-8e05-66895477704a","Type":"ContainerStarted","Data":"40dd4136b38074f14e472d81603a1c9ca73adce298851e8229769310477107b9"} Nov 02 23:01:44.828335 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:44.828279 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-ec2f9a-589bcfb455-2r9sr" podStartSLOduration=1.828260129 podStartE2EDuration="1.828260129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:01:44.827122765 +0000 UTC m=+791.065620428" watchObservedRunningTime="2026-11-02 23:01:44.828260129 +0000 UTC m=+791.066757790" Nov 02 23:01:44.950168 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:44.946336 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ec2f9a-76969d66d7-6b8dx" podStartSLOduration=1.946315244 podStartE2EDuration="1.946315244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:01:44.942417395 +0000 UTC m=+791.180915058" watchObservedRunningTime="2026-11-02 23:01:44.946315244 +0000 UTC m=+791.184812911" Nov 02 23:01:47.848867 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:47.848825 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp"] Nov 02 23:01:47.853020 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:47.852997 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp" Nov 02 23:01:47.965868 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:47.955141 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-08c2c6-serving-cert\"" Nov 02 23:01:47.965868 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:47.960410 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49503b5c-335b-4faa-b09e-4a5fa861033d-metrics-certs\") pod \"maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp\" (UID: \"49503b5c-335b-4faa-b09e-4a5fa861033d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp" Nov 02 23:01:47.965868 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:47.960482 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49503b5c-335b-4faa-b09e-4a5fa861033d-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp\" (UID: \"49503b5c-335b-4faa-b09e-4a5fa861033d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp" Nov 02 23:01:47.974250 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:47.974210 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp"] Nov 02 23:01:47.974451 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:47.974430 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh95h\" (UniqueName: \"kubernetes.io/projected/49503b5c-335b-4faa-b09e-4a5fa861033d-kube-api-access-mh95h\") pod \"maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp\" (UID: \"49503b5c-335b-4faa-b09e-4a5fa861033d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp" Nov 02 23:01:47.974544 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:47.974527 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49503b5c-335b-4faa-b09e-4a5fa861033d-maas-api-tls\") pod \"maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp\" (UID: \"49503b5c-335b-4faa-b09e-4a5fa861033d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp" Nov 02 23:01:48.076078 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:48.076044 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh95h\" (UniqueName: \"kubernetes.io/projected/49503b5c-335b-4faa-b09e-4a5fa861033d-kube-api-access-mh95h\") pod \"maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp\" (UID: \"49503b5c-335b-4faa-b09e-4a5fa861033d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp" Nov 02 23:01:48.076286 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:48.076141 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49503b5c-335b-4faa-b09e-4a5fa861033d-maas-api-tls\") pod \"maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp\" (UID: \"49503b5c-335b-4faa-b09e-4a5fa861033d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp" Nov 02 23:01:48.076286 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:48.076227 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49503b5c-335b-4faa-b09e-4a5fa861033d-metrics-certs\") pod \"maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp\" (UID: \"49503b5c-335b-4faa-b09e-4a5fa861033d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp" Nov 02 23:01:48.076286 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:01:48.076230 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-08c2c6-serving-cert: secret "maas-api-e2e-worker-w2-08c2c6-serving-cert" not found Nov 02 23:01:48.076286 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:48.076275 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49503b5c-335b-4faa-b09e-4a5fa861033d-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp\" (UID: \"49503b5c-335b-4faa-b09e-4a5fa861033d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp" Nov 02 23:01:48.076286 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:01:48.076289 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49503b5c-335b-4faa-b09e-4a5fa861033d-maas-api-tls podName:49503b5c-335b-4faa-b09e-4a5fa861033d nodeName:}" failed. No retries permitted until 2026-11-02 23:01:48.576274338 +0000 UTC m=+794.814771995 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/49503b5c-335b-4faa-b09e-4a5fa861033d-maas-api-tls") pod "maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp" (UID: "49503b5c-335b-4faa-b09e-4a5fa861033d") : secret "maas-api-e2e-worker-w2-08c2c6-serving-cert" not found Nov 02 23:01:48.077016 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:48.076995 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49503b5c-335b-4faa-b09e-4a5fa861033d-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp\" (UID: \"49503b5c-335b-4faa-b09e-4a5fa861033d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp" Nov 02 23:01:48.078972 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:48.078950 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49503b5c-335b-4faa-b09e-4a5fa861033d-metrics-certs\") pod \"maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp\" (UID: \"49503b5c-335b-4faa-b09e-4a5fa861033d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp" Nov 02 23:01:48.137076 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:48.137007 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh95h\" (UniqueName: \"kubernetes.io/projected/49503b5c-335b-4faa-b09e-4a5fa861033d-kube-api-access-mh95h\") pod \"maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp\" (UID: \"49503b5c-335b-4faa-b09e-4a5fa861033d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp" Nov 02 23:01:48.408561 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:48.408469 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-96031a-bd8c6d645-lxnf7"] Nov 02 23:01:48.412784 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:48.412763 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-96031a-bd8c6d645-lxnf7" Nov 02 23:01:48.466666 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:48.466630 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-96031a\"" Nov 02 23:01:48.466666 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:48.466630 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-96031a-dockercfg-mkpdb\"" Nov 02 23:01:48.469949 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:48.469921 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-96031a-bd8c6d645-lxnf7"] Nov 02 23:01:48.481450 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:48.481420 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/926c60cb-4000-4520-bde5-f31651560cc3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-96031a-bd8c6d645-lxnf7\" (UID: \"926c60cb-4000-4520-bde5-f31651560cc3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-96031a-bd8c6d645-lxnf7" Nov 02 23:01:48.481618 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:48.481459 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbwwt\" (UniqueName: \"kubernetes.io/projected/926c60cb-4000-4520-bde5-f31651560cc3-kube-api-access-vbwwt\") pod \"payload-processing-e2e-worker-w3-96031a-bd8c6d645-lxnf7\" (UID: \"926c60cb-4000-4520-bde5-f31651560cc3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-96031a-bd8c6d645-lxnf7" Nov 02 23:01:48.582781 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:48.582746 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49503b5c-335b-4faa-b09e-4a5fa861033d-maas-api-tls\") pod \"maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp\" (UID: \"49503b5c-335b-4faa-b09e-4a5fa861033d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp" Nov 02 23:01:48.582781 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:48.582784 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/926c60cb-4000-4520-bde5-f31651560cc3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-96031a-bd8c6d645-lxnf7\" (UID: \"926c60cb-4000-4520-bde5-f31651560cc3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-96031a-bd8c6d645-lxnf7" Nov 02 23:01:48.582999 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:48.582812 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbwwt\" (UniqueName: \"kubernetes.io/projected/926c60cb-4000-4520-bde5-f31651560cc3-kube-api-access-vbwwt\") pod \"payload-processing-e2e-worker-w3-96031a-bd8c6d645-lxnf7\" (UID: \"926c60cb-4000-4520-bde5-f31651560cc3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-96031a-bd8c6d645-lxnf7" Nov 02 23:01:48.583568 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:48.583545 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/926c60cb-4000-4520-bde5-f31651560cc3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-96031a-bd8c6d645-lxnf7\" (UID: \"926c60cb-4000-4520-bde5-f31651560cc3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-96031a-bd8c6d645-lxnf7" Nov 02 23:01:48.585362 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:48.585339 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49503b5c-335b-4faa-b09e-4a5fa861033d-maas-api-tls\") pod \"maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp\" (UID: \"49503b5c-335b-4faa-b09e-4a5fa861033d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp" Nov 02 23:01:48.643793 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:48.643760 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbwwt\" (UniqueName: \"kubernetes.io/projected/926c60cb-4000-4520-bde5-f31651560cc3-kube-api-access-vbwwt\") pod \"payload-processing-e2e-worker-w3-96031a-bd8c6d645-lxnf7\" (UID: \"926c60cb-4000-4520-bde5-f31651560cc3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-96031a-bd8c6d645-lxnf7" Nov 02 23:01:48.727726 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:48.727637 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-96031a-bd8c6d645-lxnf7" Nov 02 23:01:48.768269 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:48.768238 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp" Nov 02 23:01:49.102492 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:49.102466 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-96031a-bd8c6d645-lxnf7"] Nov 02 23:01:49.104022 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:01:49.103999 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod926c60cb_4000_4520_bde5_f31651560cc3.slice/crio-d334370fb63857a8e649e6575fa4419eb9483969366a9b373fbe44256a76220c WatchSource:0}: Error finding container d334370fb63857a8e649e6575fa4419eb9483969366a9b373fbe44256a76220c: Status 404 returned error can't find the container with id d334370fb63857a8e649e6575fa4419eb9483969366a9b373fbe44256a76220c Nov 02 23:01:49.158767 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:49.158737 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp"] Nov 02 23:01:49.160768 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:01:49.160736 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49503b5c_335b_4faa_b09e_4a5fa861033d.slice/crio-7a81a60ba1e09fd0e5d88ab78be6eabeac92271660c520fdd73b2624d0fc08de WatchSource:0}: Error finding container 7a81a60ba1e09fd0e5d88ab78be6eabeac92271660c520fdd73b2624d0fc08de: Status 404 returned error can't find the container with id 7a81a60ba1e09fd0e5d88ab78be6eabeac92271660c520fdd73b2624d0fc08de Nov 02 23:01:49.735027 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:49.734985 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp" event={"ID":"49503b5c-335b-4faa-b09e-4a5fa861033d","Type":"ContainerStarted","Data":"7a81a60ba1e09fd0e5d88ab78be6eabeac92271660c520fdd73b2624d0fc08de"} Nov 02 23:01:49.736521 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:49.736496 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-96031a-bd8c6d645-lxnf7" event={"ID":"926c60cb-4000-4520-bde5-f31651560cc3","Type":"ContainerStarted","Data":"67577f258006ded580aaf6706b757d2884522162c674fefbab9d7c2f2ec3af67"} Nov 02 23:01:49.736666 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:49.736535 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-96031a-bd8c6d645-lxnf7" event={"ID":"926c60cb-4000-4520-bde5-f31651560cc3","Type":"ContainerStarted","Data":"d334370fb63857a8e649e6575fa4419eb9483969366a9b373fbe44256a76220c"} Nov 02 23:01:49.736666 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:49.736638 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-96031a-bd8c6d645-lxnf7" Nov 02 23:01:49.853198 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:49.853103 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-96031a-bd8c6d645-lxnf7" podStartSLOduration=1.853085624 podStartE2EDuration="1.853085624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:01:49.85260393 +0000 UTC m=+796.091101608" watchObservedRunningTime="2026-11-02 23:01:49.853085624 +0000 UTC m=+796.091583289" Nov 02 23:01:50.743954 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:50.743915 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp" event={"ID":"49503b5c-335b-4faa-b09e-4a5fa861033d","Type":"ContainerStarted","Data":"840610165cb0383613f961487561402f3cf93a3694a7911630dc8c8562afca77"} Nov 02 23:01:50.744364 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:50.744115 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp" Nov 02 23:01:51.682213 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:51.682181 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-08c2c6-7bcc7fc5dc-59fbq" Nov 02 23:01:51.682381 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:51.682249 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-08c2c6-7b8bd45695-rf87d" Nov 02 23:01:51.797866 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:51.797779 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp" podStartSLOduration=3.960354174 podStartE2EDuration="4.797764388s" totalImagesPullingTime="837.410214ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:01:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:01:50.900713494 +0000 UTC m=+797.139211155" watchObservedRunningTime="2026-11-02 23:01:51.797764388 +0000 UTC m=+798.036262060" Nov 02 23:01:53.429958 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:53.429918 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc"] Nov 02 23:01:53.433912 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:53.433894 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc" Nov 02 23:01:53.485856 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:53.485821 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-ec2f9a-serving-cert\"" Nov 02 23:01:53.537826 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:53.537785 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309a0541-c36f-4914-b5b4-80b263b66f5e-metrics-certs\") pod \"maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc\" (UID: \"309a0541-c36f-4914-b5b4-80b263b66f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc" Nov 02 23:01:53.538008 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:53.537838 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/309a0541-c36f-4914-b5b4-80b263b66f5e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc\" (UID: \"309a0541-c36f-4914-b5b4-80b263b66f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc" Nov 02 23:01:53.538008 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:53.537983 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/309a0541-c36f-4914-b5b4-80b263b66f5e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc\" (UID: \"309a0541-c36f-4914-b5b4-80b263b66f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc" Nov 02 23:01:53.538166 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:53.538017 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p4w9\" (UniqueName: \"kubernetes.io/projected/309a0541-c36f-4914-b5b4-80b263b66f5e-kube-api-access-7p4w9\") pod \"maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc\" (UID: \"309a0541-c36f-4914-b5b4-80b263b66f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc" Nov 02 23:01:53.538166 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:53.538084 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc"] Nov 02 23:01:53.638668 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:53.638632 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/309a0541-c36f-4914-b5b4-80b263b66f5e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc\" (UID: \"309a0541-c36f-4914-b5b4-80b263b66f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc" Nov 02 23:01:53.638668 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:53.638675 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p4w9\" (UniqueName: \"kubernetes.io/projected/309a0541-c36f-4914-b5b4-80b263b66f5e-kube-api-access-7p4w9\") pod \"maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc\" (UID: \"309a0541-c36f-4914-b5b4-80b263b66f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc" Nov 02 23:01:53.638884 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:53.638758 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309a0541-c36f-4914-b5b4-80b263b66f5e-metrics-certs\") pod \"maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc\" (UID: \"309a0541-c36f-4914-b5b4-80b263b66f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc" Nov 02 23:01:53.638884 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:53.638789 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/309a0541-c36f-4914-b5b4-80b263b66f5e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc\" (UID: \"309a0541-c36f-4914-b5b4-80b263b66f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc" Nov 02 23:01:53.638965 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:01:53.638892 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-ec2f9a-serving-cert: secret "maas-api-e2e-worker-w0-ec2f9a-serving-cert" not found Nov 02 23:01:53.638965 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:01:53.638947 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/309a0541-c36f-4914-b5b4-80b263b66f5e-maas-api-tls podName:309a0541-c36f-4914-b5b4-80b263b66f5e nodeName:}" failed. No retries permitted until 2026-11-02 23:01:54.13893228 +0000 UTC m=+800.377429921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/309a0541-c36f-4914-b5b4-80b263b66f5e-maas-api-tls") pod "maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc" (UID: "309a0541-c36f-4914-b5b4-80b263b66f5e") : secret "maas-api-e2e-worker-w0-ec2f9a-serving-cert" not found Nov 02 23:01:53.639415 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:53.639395 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/309a0541-c36f-4914-b5b4-80b263b66f5e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc\" (UID: \"309a0541-c36f-4914-b5b4-80b263b66f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc" Nov 02 23:01:53.641404 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:53.641382 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309a0541-c36f-4914-b5b4-80b263b66f5e-metrics-certs\") pod \"maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc\" (UID: \"309a0541-c36f-4914-b5b4-80b263b66f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc" Nov 02 23:01:53.700778 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:53.700698 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p4w9\" (UniqueName: \"kubernetes.io/projected/309a0541-c36f-4914-b5b4-80b263b66f5e-kube-api-access-7p4w9\") pod \"maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc\" (UID: \"309a0541-c36f-4914-b5b4-80b263b66f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc" Nov 02 23:01:54.144827 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:54.144792 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/309a0541-c36f-4914-b5b4-80b263b66f5e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc\" (UID: \"309a0541-c36f-4914-b5b4-80b263b66f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc" Nov 02 23:01:54.147554 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:54.147528 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/309a0541-c36f-4914-b5b4-80b263b66f5e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc\" (UID: \"309a0541-c36f-4914-b5b4-80b263b66f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc" Nov 02 23:01:54.347846 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:54.347807 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc" Nov 02 23:01:54.728916 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:54.728887 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc"] Nov 02 23:01:54.730376 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:01:54.730351 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod309a0541_c36f_4914_b5b4_80b263b66f5e.slice/crio-ded9c02bb8b404e46e67b2dd72f3564866f085d9326cb9e2a27616761ce2b725 WatchSource:0}: Error finding container ded9c02bb8b404e46e67b2dd72f3564866f085d9326cb9e2a27616761ce2b725: Status 404 returned error can't find the container with id ded9c02bb8b404e46e67b2dd72f3564866f085d9326cb9e2a27616761ce2b725 Nov 02 23:01:54.772240 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:54.772199 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc" event={"ID":"309a0541-c36f-4914-b5b4-80b263b66f5e","Type":"ContainerStarted","Data":"ded9c02bb8b404e46e67b2dd72f3564866f085d9326cb9e2a27616761ce2b725"} Nov 02 23:01:55.712866 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:55.712828 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-ec2f9a-589bcfb455-2r9sr" Nov 02 23:01:55.713030 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:55.713015 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ec2f9a-76969d66d7-6b8dx" Nov 02 23:01:55.781079 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:55.781038 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc" event={"ID":"309a0541-c36f-4914-b5b4-80b263b66f5e","Type":"ContainerStarted","Data":"c55917c0200acd443038cad7afdd58affca936b53bf78879137c6f221694d64b"} Nov 02 23:01:55.781521 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:55.781152 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc" Nov 02 23:01:56.061353 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:56.061298 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc" podStartSLOduration=2.2557036679999998 podStartE2EDuration="3.061283996s" totalImagesPullingTime="805.580328ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:01:56.060253525 +0000 UTC m=+802.298751215" watchObservedRunningTime="2026-11-02 23:01:56.061283996 +0000 UTC m=+802.299781655" Nov 02 23:01:56.760153 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:01:56.759224 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp" Nov 02 23:01:57.844990 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:01:57.844956 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:02:00.746783 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:00.746757 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-96031a-bd8c6d645-lxnf7" Nov 02 23:02:01.794251 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:01.794225 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc" Nov 02 23:02:02.961769 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:02.961732 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq"] Nov 02 23:02:02.962194 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:02.961931 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq" podUID="73d19747-5f1e-459c-81f7-cf4eefc86b40" containerName="maas-api" containerID="cri-o://d2c71e006d37c8cb90a6cffc2c9acd3f23e406bea8edc19df2b172149aa153ad" gracePeriod=30 Nov 02 23:02:03.231254 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:03.231229 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq" Nov 02 23:02:03.334751 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:03.334705 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73d19747-5f1e-459c-81f7-cf4eefc86b40-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d19747-5f1e-459c-81f7-cf4eefc86b40-openshift-service-ca\") pod \"73d19747-5f1e-459c-81f7-cf4eefc86b40\" (UID: \"73d19747-5f1e-459c-81f7-cf4eefc86b40\") " Nov 02 23:02:03.334751 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:03.334740 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73d19747-5f1e-459c-81f7-cf4eefc86b40-metrics-certs\" (UniqueName: \"kubernetes.io/secret/73d19747-5f1e-459c-81f7-cf4eefc86b40-metrics-certs\") pod \"73d19747-5f1e-459c-81f7-cf4eefc86b40\" (UID: \"73d19747-5f1e-459c-81f7-cf4eefc86b40\") " Nov 02 23:02:03.335092 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:03.334907 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73d19747-5f1e-459c-81f7-cf4eefc86b40-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73d19747-5f1e-459c-81f7-cf4eefc86b40-maas-api-tls\") pod \"73d19747-5f1e-459c-81f7-cf4eefc86b40\" (UID: \"73d19747-5f1e-459c-81f7-cf4eefc86b40\") " Nov 02 23:02:03.335092 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:03.334948 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73d19747-5f1e-459c-81f7-cf4eefc86b40-kube-api-access-9kdz2\" (UniqueName: \"kubernetes.io/projected/73d19747-5f1e-459c-81f7-cf4eefc86b40-kube-api-access-9kdz2\") pod \"73d19747-5f1e-459c-81f7-cf4eefc86b40\" (UID: \"73d19747-5f1e-459c-81f7-cf4eefc86b40\") " Nov 02 23:02:03.335235 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:03.335178 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73d19747-5f1e-459c-81f7-cf4eefc86b40-openshift-service-ca" pod "73d19747-5f1e-459c-81f7-cf4eefc86b40" (UID: "73d19747-5f1e-459c-81f7-cf4eefc86b40"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:02:03.337527 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:03.337475 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73d19747-5f1e-459c-81f7-cf4eefc86b40-metrics-certs" pod "73d19747-5f1e-459c-81f7-cf4eefc86b40" (UID: "73d19747-5f1e-459c-81f7-cf4eefc86b40"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:02:03.337840 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:03.337815 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73d19747-5f1e-459c-81f7-cf4eefc86b40-kube-api-access-9kdz2" pod "73d19747-5f1e-459c-81f7-cf4eefc86b40" (UID: "73d19747-5f1e-459c-81f7-cf4eefc86b40"). InnerVolumeSpecName "kube-api-access-9kdz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:02:03.339540 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:03.339510 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73d19747-5f1e-459c-81f7-cf4eefc86b40-maas-api-tls" pod "73d19747-5f1e-459c-81f7-cf4eefc86b40" (UID: "73d19747-5f1e-459c-81f7-cf4eefc86b40"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:02:03.436470 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:03.436440 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73d19747-5f1e-459c-81f7-cf4eefc86b40-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:02:03.436470 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:03.436468 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kdz2\" (UniqueName: \"kubernetes.io/projected/73d19747-5f1e-459c-81f7-cf4eefc86b40-kube-api-access-9kdz2\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:02:03.436666 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:03.436479 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d19747-5f1e-459c-81f7-cf4eefc86b40-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:02:03.436666 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:03.436489 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73d19747-5f1e-459c-81f7-cf4eefc86b40-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:02:03.832410 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:03.832373 5727 generic.go:356] "Generic (PLEG): container finished" podID="73d19747-5f1e-459c-81f7-cf4eefc86b40" containerID="d2c71e006d37c8cb90a6cffc2c9acd3f23e406bea8edc19df2b172149aa153ad" exitCode=0 Nov 02 23:02:03.832410 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:03.832412 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq" Nov 02 23:02:03.832652 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:03.832449 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq" event={"ID":"73d19747-5f1e-459c-81f7-cf4eefc86b40","Type":"ContainerDied","Data":"d2c71e006d37c8cb90a6cffc2c9acd3f23e406bea8edc19df2b172149aa153ad"} Nov 02 23:02:03.832652 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:03.832507 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq" event={"ID":"73d19747-5f1e-459c-81f7-cf4eefc86b40","Type":"ContainerDied","Data":"1c47eaf6a9b4996418491eccb0364e66014bd965d6abd9d0668c115b83c77e92"} Nov 02 23:02:03.832652 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:03.832532 5727 scope.go:122] "RemoveContainer" containerID="d2c71e006d37c8cb90a6cffc2c9acd3f23e406bea8edc19df2b172149aa153ad" Nov 02 23:02:03.844302 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:03.844284 5727 scope.go:122] "RemoveContainer" containerID="d2c71e006d37c8cb90a6cffc2c9acd3f23e406bea8edc19df2b172149aa153ad" Nov 02 23:02:03.844565 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:02:03.844549 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2c71e006d37c8cb90a6cffc2c9acd3f23e406bea8edc19df2b172149aa153ad\": container with ID starting with d2c71e006d37c8cb90a6cffc2c9acd3f23e406bea8edc19df2b172149aa153ad not found: ID does not exist" containerID="d2c71e006d37c8cb90a6cffc2c9acd3f23e406bea8edc19df2b172149aa153ad" Nov 02 23:02:03.844607 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:03.844574 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2c71e006d37c8cb90a6cffc2c9acd3f23e406bea8edc19df2b172149aa153ad"} err="failed to get container status \"d2c71e006d37c8cb90a6cffc2c9acd3f23e406bea8edc19df2b172149aa153ad\": rpc error: code = NotFound desc = could not find container \"d2c71e006d37c8cb90a6cffc2c9acd3f23e406bea8edc19df2b172149aa153ad\": container with ID starting with d2c71e006d37c8cb90a6cffc2c9acd3f23e406bea8edc19df2b172149aa153ad not found: ID does not exist" Nov 02 23:02:04.002520 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:04.002489 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq"] Nov 02 23:02:04.008209 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:04.008181 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-708f46a0-74b4cb4b6b-qw8cq"] Nov 02 23:02:05.883608 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:05.883570 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73d19747-5f1e-459c-81f7-cf4eefc86b40" path="/var/lib/kubelet/pods/73d19747-5f1e-459c-81f7-cf4eefc86b40/volumes" Nov 02 23:02:14.500518 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:14.500436 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b42983-f5cc57d49-hbnxp"] Nov 02 23:02:14.500995 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:14.500981 5727 cpu_manager.go:414] "containerMap: removing container" podUID="73d19747-5f1e-459c-81f7-cf4eefc86b40" containerName="maas-api" Nov 02 23:02:14.501039 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:14.500996 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73d19747-5f1e-459c-81f7-cf4eefc86b40" containerName="maas-api" Nov 02 23:02:14.501118 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:14.501108 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="73d19747-5f1e-459c-81f7-cf4eefc86b40" containerName="maas-api" Nov 02 23:02:14.506031 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:14.506007 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b42983-f5cc57d49-hbnxp" Nov 02 23:02:14.541691 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:14.541664 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3472bbbd-3579-4f37-a8f0-48fcbe1cb627-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b42983-f5cc57d49-hbnxp\" (UID: \"3472bbbd-3579-4f37-a8f0-48fcbe1cb627\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b42983-f5cc57d49-hbnxp" Nov 02 23:02:14.541803 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:14.541732 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrrf7\" (UniqueName: \"kubernetes.io/projected/3472bbbd-3579-4f37-a8f0-48fcbe1cb627-kube-api-access-wrrf7\") pod \"payload-processing-e2e-worker-w1-b42983-f5cc57d49-hbnxp\" (UID: \"3472bbbd-3579-4f37-a8f0-48fcbe1cb627\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b42983-f5cc57d49-hbnxp" Nov 02 23:02:14.608357 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:14.608324 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-b42983\"" Nov 02 23:02:14.608527 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:14.608363 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-b42983-dockercfg-dmks8\"" Nov 02 23:02:14.611010 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:14.610988 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b42983-f5cc57d49-hbnxp"] Nov 02 23:02:14.642550 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:14.642527 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3472bbbd-3579-4f37-a8f0-48fcbe1cb627-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b42983-f5cc57d49-hbnxp\" (UID: \"3472bbbd-3579-4f37-a8f0-48fcbe1cb627\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b42983-f5cc57d49-hbnxp" Nov 02 23:02:14.642693 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:14.642566 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrrf7\" (UniqueName: \"kubernetes.io/projected/3472bbbd-3579-4f37-a8f0-48fcbe1cb627-kube-api-access-wrrf7\") pod \"payload-processing-e2e-worker-w1-b42983-f5cc57d49-hbnxp\" (UID: \"3472bbbd-3579-4f37-a8f0-48fcbe1cb627\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b42983-f5cc57d49-hbnxp" Nov 02 23:02:14.643265 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:14.643244 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3472bbbd-3579-4f37-a8f0-48fcbe1cb627-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b42983-f5cc57d49-hbnxp\" (UID: \"3472bbbd-3579-4f37-a8f0-48fcbe1cb627\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b42983-f5cc57d49-hbnxp" Nov 02 23:02:14.703387 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:14.703355 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrrf7\" (UniqueName: \"kubernetes.io/projected/3472bbbd-3579-4f37-a8f0-48fcbe1cb627-kube-api-access-wrrf7\") pod \"payload-processing-e2e-worker-w1-b42983-f5cc57d49-hbnxp\" (UID: \"3472bbbd-3579-4f37-a8f0-48fcbe1cb627\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b42983-f5cc57d49-hbnxp" Nov 02 23:02:14.819804 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:14.819769 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b42983-f5cc57d49-hbnxp" Nov 02 23:02:15.196621 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:15.196584 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b42983-f5cc57d49-hbnxp"] Nov 02 23:02:15.197815 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:02:15.197791 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3472bbbd_3579_4f37_a8f0_48fcbe1cb627.slice/crio-8ff7cd1e84e4ad9041ae85360c5d9420b3413f3ac4469a963968d3a0d6db7b41 WatchSource:0}: Error finding container 8ff7cd1e84e4ad9041ae85360c5d9420b3413f3ac4469a963968d3a0d6db7b41: Status 404 returned error can't find the container with id 8ff7cd1e84e4ad9041ae85360c5d9420b3413f3ac4469a963968d3a0d6db7b41 Nov 02 23:02:15.217569 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:15.217548 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-637fdb-97f6995c-szpck"] Nov 02 23:02:15.222376 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:15.222352 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-637fdb-97f6995c-szpck" Nov 02 23:02:15.248760 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:15.248733 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqqmr\" (UniqueName: \"kubernetes.io/projected/5eff97c2-fd05-445f-bc27-2d5f56df0f09-kube-api-access-hqqmr\") pod \"maas-api-e2e-worker-w4-637fdb-97f6995c-szpck\" (UID: \"5eff97c2-fd05-445f-bc27-2d5f56df0f09\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-637fdb-97f6995c-szpck" Nov 02 23:02:15.248869 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:15.248798 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eff97c2-fd05-445f-bc27-2d5f56df0f09-metrics-certs\") pod \"maas-api-e2e-worker-w4-637fdb-97f6995c-szpck\" (UID: \"5eff97c2-fd05-445f-bc27-2d5f56df0f09\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-637fdb-97f6995c-szpck" Nov 02 23:02:15.248958 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:15.248937 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eff97c2-fd05-445f-bc27-2d5f56df0f09-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-637fdb-97f6995c-szpck\" (UID: \"5eff97c2-fd05-445f-bc27-2d5f56df0f09\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-637fdb-97f6995c-szpck" Nov 02 23:02:15.249016 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:15.248978 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eff97c2-fd05-445f-bc27-2d5f56df0f09-maas-api-tls\") pod \"maas-api-e2e-worker-w4-637fdb-97f6995c-szpck\" (UID: \"5eff97c2-fd05-445f-bc27-2d5f56df0f09\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-637fdb-97f6995c-szpck" Nov 02 23:02:15.280764 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:15.280738 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-637fdb-serving-cert\"" Nov 02 23:02:15.282059 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:15.282036 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-637fdb-97f6995c-szpck"] Nov 02 23:02:15.349582 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:15.349553 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqqmr\" (UniqueName: \"kubernetes.io/projected/5eff97c2-fd05-445f-bc27-2d5f56df0f09-kube-api-access-hqqmr\") pod \"maas-api-e2e-worker-w4-637fdb-97f6995c-szpck\" (UID: \"5eff97c2-fd05-445f-bc27-2d5f56df0f09\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-637fdb-97f6995c-szpck" Nov 02 23:02:15.349582 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:15.349597 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eff97c2-fd05-445f-bc27-2d5f56df0f09-metrics-certs\") pod \"maas-api-e2e-worker-w4-637fdb-97f6995c-szpck\" (UID: \"5eff97c2-fd05-445f-bc27-2d5f56df0f09\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-637fdb-97f6995c-szpck" Nov 02 23:02:15.349856 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:15.349772 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eff97c2-fd05-445f-bc27-2d5f56df0f09-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-637fdb-97f6995c-szpck\" (UID: \"5eff97c2-fd05-445f-bc27-2d5f56df0f09\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-637fdb-97f6995c-szpck" Nov 02 23:02:15.349856 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:15.349838 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eff97c2-fd05-445f-bc27-2d5f56df0f09-maas-api-tls\") pod \"maas-api-e2e-worker-w4-637fdb-97f6995c-szpck\" (UID: \"5eff97c2-fd05-445f-bc27-2d5f56df0f09\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-637fdb-97f6995c-szpck" Nov 02 23:02:15.349989 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:02:15.349972 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-637fdb-serving-cert: secret "maas-api-e2e-worker-w4-637fdb-serving-cert" not found Nov 02 23:02:15.350055 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:02:15.350024 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eff97c2-fd05-445f-bc27-2d5f56df0f09-maas-api-tls podName:5eff97c2-fd05-445f-bc27-2d5f56df0f09 nodeName:}" failed. No retries permitted until 2026-11-02 23:02:15.850010848 +0000 UTC m=+822.088508488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5eff97c2-fd05-445f-bc27-2d5f56df0f09-maas-api-tls") pod "maas-api-e2e-worker-w4-637fdb-97f6995c-szpck" (UID: "5eff97c2-fd05-445f-bc27-2d5f56df0f09") : secret "maas-api-e2e-worker-w4-637fdb-serving-cert" not found Nov 02 23:02:15.350591 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:15.350570 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eff97c2-fd05-445f-bc27-2d5f56df0f09-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-637fdb-97f6995c-szpck\" (UID: \"5eff97c2-fd05-445f-bc27-2d5f56df0f09\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-637fdb-97f6995c-szpck" Nov 02 23:02:15.352472 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:15.352451 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eff97c2-fd05-445f-bc27-2d5f56df0f09-metrics-certs\") pod \"maas-api-e2e-worker-w4-637fdb-97f6995c-szpck\" (UID: \"5eff97c2-fd05-445f-bc27-2d5f56df0f09\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-637fdb-97f6995c-szpck" Nov 02 23:02:15.410483 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:15.410451 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqqmr\" (UniqueName: \"kubernetes.io/projected/5eff97c2-fd05-445f-bc27-2d5f56df0f09-kube-api-access-hqqmr\") pod \"maas-api-e2e-worker-w4-637fdb-97f6995c-szpck\" (UID: \"5eff97c2-fd05-445f-bc27-2d5f56df0f09\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-637fdb-97f6995c-szpck" Nov 02 23:02:15.854267 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:15.854229 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eff97c2-fd05-445f-bc27-2d5f56df0f09-maas-api-tls\") pod \"maas-api-e2e-worker-w4-637fdb-97f6995c-szpck\" (UID: \"5eff97c2-fd05-445f-bc27-2d5f56df0f09\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-637fdb-97f6995c-szpck" Nov 02 23:02:15.856953 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:15.856920 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eff97c2-fd05-445f-bc27-2d5f56df0f09-maas-api-tls\") pod \"maas-api-e2e-worker-w4-637fdb-97f6995c-szpck\" (UID: \"5eff97c2-fd05-445f-bc27-2d5f56df0f09\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-637fdb-97f6995c-szpck" Nov 02 23:02:15.915937 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:15.915910 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b42983-f5cc57d49-hbnxp" event={"ID":"3472bbbd-3579-4f37-a8f0-48fcbe1cb627","Type":"ContainerStarted","Data":"3a5aacdce3f729603ead20ef22bf908a8fa99baf403cd81d70c2fbb9b1499b14"} Nov 02 23:02:15.916062 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:15.915966 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b42983-f5cc57d49-hbnxp" event={"ID":"3472bbbd-3579-4f37-a8f0-48fcbe1cb627","Type":"ContainerStarted","Data":"8ff7cd1e84e4ad9041ae85360c5d9420b3413f3ac4469a963968d3a0d6db7b41"} Nov 02 23:02:15.916166 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:15.916104 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-b42983-f5cc57d49-hbnxp" Nov 02 23:02:16.031642 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:16.031572 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-b42983-f5cc57d49-hbnxp" podStartSLOduration=2.031554161 podStartE2EDuration="2.031554161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:02:16.030305513 +0000 UTC m=+822.268803218" watchObservedRunningTime="2026-11-02 23:02:16.031554161 +0000 UTC m=+822.270051824" Nov 02 23:02:16.156888 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:16.156811 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-637fdb-97f6995c-szpck" Nov 02 23:02:16.541288 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:16.541267 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-637fdb-97f6995c-szpck"] Nov 02 23:02:16.543078 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:02:16.543056 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eff97c2_fd05_445f_bc27_2d5f56df0f09.slice/crio-0f7aaad57c8d8a9b01dec90471d081d326a69ba07711025c733d0526333d2407 WatchSource:0}: Error finding container 0f7aaad57c8d8a9b01dec90471d081d326a69ba07711025c733d0526333d2407: Status 404 returned error can't find the container with id 0f7aaad57c8d8a9b01dec90471d081d326a69ba07711025c733d0526333d2407 Nov 02 23:02:16.922955 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:16.922911 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-637fdb-97f6995c-szpck" event={"ID":"5eff97c2-fd05-445f-bc27-2d5f56df0f09","Type":"ContainerStarted","Data":"0f7aaad57c8d8a9b01dec90471d081d326a69ba07711025c733d0526333d2407"} Nov 02 23:02:17.931414 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:17.931380 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-637fdb-97f6995c-szpck" event={"ID":"5eff97c2-fd05-445f-bc27-2d5f56df0f09","Type":"ContainerStarted","Data":"a8d9bf24fe0afb3ae71297954975aaa18a675ab974e76dc3c602a33907d0e89a"} Nov 02 23:02:17.931819 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:17.931522 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-637fdb-97f6995c-szpck" Nov 02 23:02:18.050205 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:18.050147 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-637fdb-97f6995c-szpck" podStartSLOduration=2.183720681 podStartE2EDuration="3.050109869s" totalImagesPullingTime="866.389188ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:02:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:02:18.047964517 +0000 UTC m=+824.286462190" watchObservedRunningTime="2026-11-02 23:02:18.050109869 +0000 UTC m=+824.288607527" Nov 02 23:02:23.944581 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:23.944554 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-637fdb-97f6995c-szpck" Nov 02 23:02:26.926697 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:26.926669 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-b42983-f5cc57d49-hbnxp" Nov 02 23:02:27.844727 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:02:27.844694 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:02:36.723267 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:36.723231 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc"] Nov 02 23:02:36.728926 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:36.728903 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc" Nov 02 23:02:36.831564 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:36.831534 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-b42983-serving-cert\"" Nov 02 23:02:36.834623 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:36.834600 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc"] Nov 02 23:02:36.866466 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:36.866436 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13206437-6c0e-4cbe-9f87-ce696dc65d6f-metrics-certs\") pod \"maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc\" (UID: \"13206437-6c0e-4cbe-9f87-ce696dc65d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc" Nov 02 23:02:36.866629 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:36.866600 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13206437-6c0e-4cbe-9f87-ce696dc65d6f-maas-api-tls\") pod \"maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc\" (UID: \"13206437-6c0e-4cbe-9f87-ce696dc65d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc" Nov 02 23:02:36.866709 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:36.866686 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcnqp\" (UniqueName: \"kubernetes.io/projected/13206437-6c0e-4cbe-9f87-ce696dc65d6f-kube-api-access-jcnqp\") pod \"maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc\" (UID: \"13206437-6c0e-4cbe-9f87-ce696dc65d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc" Nov 02 23:02:36.866759 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:36.866720 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13206437-6c0e-4cbe-9f87-ce696dc65d6f-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc\" (UID: \"13206437-6c0e-4cbe-9f87-ce696dc65d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc" Nov 02 23:02:36.967696 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:36.967647 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13206437-6c0e-4cbe-9f87-ce696dc65d6f-metrics-certs\") pod \"maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc\" (UID: \"13206437-6c0e-4cbe-9f87-ce696dc65d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc" Nov 02 23:02:36.968012 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:36.967791 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13206437-6c0e-4cbe-9f87-ce696dc65d6f-maas-api-tls\") pod \"maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc\" (UID: \"13206437-6c0e-4cbe-9f87-ce696dc65d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc" Nov 02 23:02:36.968012 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:36.967827 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcnqp\" (UniqueName: \"kubernetes.io/projected/13206437-6c0e-4cbe-9f87-ce696dc65d6f-kube-api-access-jcnqp\") pod \"maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc\" (UID: \"13206437-6c0e-4cbe-9f87-ce696dc65d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc" Nov 02 23:02:36.968012 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:36.967854 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13206437-6c0e-4cbe-9f87-ce696dc65d6f-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc\" (UID: \"13206437-6c0e-4cbe-9f87-ce696dc65d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc" Nov 02 23:02:36.968503 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:02:36.968480 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42983-serving-cert: secret "maas-api-e2e-worker-w1-b42983-serving-cert" not found Nov 02 23:02:36.969308 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:02:36.969286 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13206437-6c0e-4cbe-9f87-ce696dc65d6f-maas-api-tls podName:13206437-6c0e-4cbe-9f87-ce696dc65d6f nodeName:}" failed. No retries permitted until 2026-11-02 23:02:37.469262149 +0000 UTC m=+843.707759793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/13206437-6c0e-4cbe-9f87-ce696dc65d6f-maas-api-tls") pod "maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc" (UID: "13206437-6c0e-4cbe-9f87-ce696dc65d6f") : secret "maas-api-e2e-worker-w1-b42983-serving-cert" not found Nov 02 23:02:36.969456 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:36.969350 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13206437-6c0e-4cbe-9f87-ce696dc65d6f-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc\" (UID: \"13206437-6c0e-4cbe-9f87-ce696dc65d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc" Nov 02 23:02:36.971944 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:36.971921 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13206437-6c0e-4cbe-9f87-ce696dc65d6f-metrics-certs\") pod \"maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc\" (UID: \"13206437-6c0e-4cbe-9f87-ce696dc65d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc" Nov 02 23:02:37.028757 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:37.028699 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcnqp\" (UniqueName: \"kubernetes.io/projected/13206437-6c0e-4cbe-9f87-ce696dc65d6f-kube-api-access-jcnqp\") pod \"maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc\" (UID: \"13206437-6c0e-4cbe-9f87-ce696dc65d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc" Nov 02 23:02:37.474992 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:37.474960 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13206437-6c0e-4cbe-9f87-ce696dc65d6f-maas-api-tls\") pod \"maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc\" (UID: \"13206437-6c0e-4cbe-9f87-ce696dc65d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc" Nov 02 23:02:37.477846 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:37.477822 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13206437-6c0e-4cbe-9f87-ce696dc65d6f-maas-api-tls\") pod \"maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc\" (UID: \"13206437-6c0e-4cbe-9f87-ce696dc65d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc" Nov 02 23:02:37.641872 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:37.641836 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc" Nov 02 23:02:38.030719 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:38.030696 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc"] Nov 02 23:02:38.032232 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:02:38.032208 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13206437_6c0e_4cbe_9f87_ce696dc65d6f.slice/crio-187cdbfaacf7f3191c2a1913e508c310ad37fd50f5c56127ba46ee2dbf8c448c WatchSource:0}: Error finding container 187cdbfaacf7f3191c2a1913e508c310ad37fd50f5c56127ba46ee2dbf8c448c: Status 404 returned error can't find the container with id 187cdbfaacf7f3191c2a1913e508c310ad37fd50f5c56127ba46ee2dbf8c448c Nov 02 23:02:38.068421 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:38.068394 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc" event={"ID":"13206437-6c0e-4cbe-9f87-ce696dc65d6f","Type":"ContainerStarted","Data":"187cdbfaacf7f3191c2a1913e508c310ad37fd50f5c56127ba46ee2dbf8c448c"} Nov 02 23:02:39.076547 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:39.076510 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc" event={"ID":"13206437-6c0e-4cbe-9f87-ce696dc65d6f","Type":"ContainerStarted","Data":"83fcb42d8e9bd3c1882e84987a4b85dff9af8ed08444fab11545fed9a19727ec"} Nov 02 23:02:39.076993 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:39.076622 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc" Nov 02 23:02:39.196543 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:39.196445 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc" podStartSLOduration=2.367388258 podStartE2EDuration="3.19643122s" totalImagesPullingTime="829.042962ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:02:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:02:39.194985665 +0000 UTC m=+845.433483339" watchObservedRunningTime="2026-11-02 23:02:39.19643122 +0000 UTC m=+845.434928943" Nov 02 23:02:45.088319 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:45.088291 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc" Nov 02 23:02:52.046087 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:52.046057 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs"] Nov 02 23:02:52.049982 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:52.049966 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:02:52.102247 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:52.102215 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b42983-openshift-default-dockercfg-xchbk\"" Nov 02 23:02:52.229540 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:52.229508 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:02:52.229718 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:52.229552 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:02:52.229718 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:52.229584 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dg22\" (UniqueName: \"kubernetes.io/projected/c2048b92-e5d0-4a0c-9951-893ecc868e9b-kube-api-access-8dg22\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:02:52.229718 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:52.229707 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:02:52.229853 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:52.229745 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:02:52.229853 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:52.229773 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:02:52.229853 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:52.229804 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:02:52.229948 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:52.229862 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:02:52.229948 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:52.229895 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:02:52.251032 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:52.251005 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs"] Nov 02 23:02:52.331390 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:52.331359 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:02:52.331564 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:52.331402 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:02:52.331564 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:52.331419 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:02:52.331564 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:52.331459 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:02:52.331564 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:52.331506 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:02:52.331564 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:52.331543 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:02:52.331824 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:52.331586 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:02:52.331824 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:52.331621 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dg22\" (UniqueName: \"kubernetes.io/projected/c2048b92-e5d0-4a0c-9951-893ecc868e9b-kube-api-access-8dg22\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:02:52.331923 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:52.331896 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:02:52.331984 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:52.331939 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:02:52.331984 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:52.331962 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:02:52.332091 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:52.332005 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:02:52.332091 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:52.332018 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:02:52.332222 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:52.332170 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:02:52.334046 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:52.334020 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:02:52.334373 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:52.334354 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:02:52.392415 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:52.392389 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:02:52.392631 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:52.392605 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dg22\" (UniqueName: \"kubernetes.io/projected/c2048b92-e5d0-4a0c-9951-893ecc868e9b-kube-api-access-8dg22\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:02:52.669185 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:52.669069 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:02:53.105054 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:02:53.105023 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2048b92_e5d0_4a0c_9951_893ecc868e9b.slice/crio-43d736d4d831242da173fe6d4f379eec5f5d2f28f908086f72cad50603b4328c WatchSource:0}: Error finding container 43d736d4d831242da173fe6d4f379eec5f5d2f28f908086f72cad50603b4328c: Status 404 returned error can't find the container with id 43d736d4d831242da173fe6d4f379eec5f5d2f28f908086f72cad50603b4328c Nov 02 23:02:53.107529 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:53.107491 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:02:53.107634 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:53.107561 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:02:53.107634 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:53.107588 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:02:53.119175 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:53.119152 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs"] Nov 02 23:02:53.168679 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:53.168650 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" event={"ID":"c2048b92-e5d0-4a0c-9951-893ecc868e9b","Type":"ContainerStarted","Data":"43d736d4d831242da173fe6d4f379eec5f5d2f28f908086f72cad50603b4328c"} Nov 02 23:02:54.188308 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:54.188265 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" event={"ID":"c2048b92-e5d0-4a0c-9951-893ecc868e9b","Type":"ContainerStarted","Data":"591e5bd02b8b9f7f9efd03a4be454b40b6746243e11d2dfc91a5c65734532d60"} Nov 02 23:02:54.375506 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:54.375440 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" podStartSLOduration=3.375419445 podStartE2EDuration="3.375419445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:02:54.372233096 +0000 UTC m=+860.610730756" watchObservedRunningTime="2026-11-02 23:02:54.375419445 +0000 UTC m=+860.613917111" Nov 02 23:02:54.675154 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:54.670191 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:02:54.675154 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:54.671893 5727 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 02 23:02:54.675154 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:54.671942 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" podUID="c2048b92-e5d0-4a0c-9951-893ecc868e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 02 23:02:55.676058 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:55.675410 5727 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 02 23:02:55.676058 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:55.675490 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" podUID="c2048b92-e5d0-4a0c-9951-893ecc868e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 02 23:02:56.669526 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:56.669487 5727 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 02 23:02:56.669845 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:56.669812 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" podUID="c2048b92-e5d0-4a0c-9951-893ecc868e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 02 23:02:57.671154 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:57.670194 5727 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 02 23:02:57.671154 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:57.670264 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" podUID="c2048b92-e5d0-4a0c-9951-893ecc868e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 02 23:02:57.891805 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:02:57.891725 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:02:58.675157 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:58.675008 5727 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 02 23:02:58.675157 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:58.675074 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" podUID="c2048b92-e5d0-4a0c-9951-893ecc868e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 02 23:02:59.673457 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:59.673430 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:02:59.673863 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:59.673838 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:02:59.674555 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:02:59.674534 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:03:22.281119 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:22.281086 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv"] Nov 02 23:03:22.285047 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:22.285028 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:22.391411 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:22.391379 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv"] Nov 02 23:03:22.394109 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:22.394081 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:22.394271 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:22.394119 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:22.394271 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:22.394257 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:22.394375 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:22.394306 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:22.394423 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:22.394388 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwl59\" (UniqueName: \"kubernetes.io/projected/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-kube-api-access-gwl59\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:22.394483 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:22.394468 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:22.394526 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:22.394498 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:22.394612 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:22.394591 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:22.394725 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:22.394710 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:22.496465 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:22.496428 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:22.496639 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:22.496484 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:22.496639 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:22.496537 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:22.496639 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:22.496594 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:22.496826 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:22.496646 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:22.496826 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:22.496737 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:22.497263 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:22.497229 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:22.497413 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:22.496901 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:22.497413 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:22.497306 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:22.497413 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:22.497370 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:22.497413 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:22.497381 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:22.497413 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:22.497400 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwl59\" (UniqueName: \"kubernetes.io/projected/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-kube-api-access-gwl59\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:22.497723 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:22.497431 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:22.498012 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:22.497990 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:22.499331 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:22.499305 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:22.500183 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:22.500161 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:22.557226 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:22.557172 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:22.557344 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:22.557331 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwl59\" (UniqueName: \"kubernetes.io/projected/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-kube-api-access-gwl59\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:22.599013 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:22.598985 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:23.027766 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:23.027743 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv"] Nov 02 23:03:23.034753 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:23.034714 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:03:23.034858 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:23.034804 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:03:23.034858 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:23.034847 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:03:23.412945 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:23.412899 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" event={"ID":"3bd21a3d-ad0a-48db-80fa-37b40e05bb74","Type":"ContainerStarted","Data":"9009a2c0770b86b983910d054034e182e5ee9b02eec7b2b7fe2c7894f7ba53f8"} Nov 02 23:03:23.413349 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:23.412970 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" event={"ID":"3bd21a3d-ad0a-48db-80fa-37b40e05bb74","Type":"ContainerStarted","Data":"2a61943e6e8047a178d7092b02af939a6a995a661ee0d931cce471e9fd4681ce"} Nov 02 23:03:23.558047 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:23.557971 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" podStartSLOduration=1.557948736 podStartE2EDuration="1.557948736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:03:23.548078425 +0000 UTC m=+889.786576106" watchObservedRunningTime="2026-11-02 23:03:23.557948736 +0000 UTC m=+889.796446401" Nov 02 23:03:23.599898 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:23.599862 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:24.601000 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:24.600970 5727 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 02 23:03:24.601450 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:24.601020 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" podUID="3bd21a3d-ad0a-48db-80fa-37b40e05bb74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 02 23:03:25.599518 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:25.599479 5727 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 02 23:03:25.599790 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:25.599765 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" podUID="3bd21a3d-ad0a-48db-80fa-37b40e05bb74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 02 23:03:26.602012 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:26.601965 5727 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 02 23:03:26.602462 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:26.602036 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" podUID="3bd21a3d-ad0a-48db-80fa-37b40e05bb74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 02 23:03:27.599724 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:27.599686 5727 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 02 23:03:27.599912 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:27.599738 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" podUID="3bd21a3d-ad0a-48db-80fa-37b40e05bb74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 02 23:03:27.850207 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:03:27.849454 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:03:28.599366 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:28.599332 5727 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 02 23:03:28.599626 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:28.599605 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" podUID="3bd21a3d-ad0a-48db-80fa-37b40e05bb74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 02 23:03:29.604514 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:29.604485 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:29.604986 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:29.604977 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:29.605738 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:29.605718 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:03:34.220201 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:34.220171 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 02 23:03:34.237050 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:34.237024 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 23:03:34.239531 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:34.239511 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 02 23:03:34.264363 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:03:34.264330 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 23:03:57.845611 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:03:57.845575 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:04:27.845360 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:04:27.845321 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:04:30.815559 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:30.815508 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz"] Nov 02 23:04:30.819685 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:30.819659 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" Nov 02 23:04:30.923794 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:30.923749 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-637fdb\"/\"kube-root-ca.crt\"" Nov 02 23:04:30.923794 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:30.923754 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-637fdb\"/\"default-dockercfg-6ffdh\"" Nov 02 23:04:30.924008 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:30.923806 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-637fdb\"/\"premium-w4-637fdb-kserve-self-signed-certs\"" Nov 02 23:04:30.924008 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:30.923829 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-637fdb\"/\"openshift-service-ca.crt\"" Nov 02 23:04:30.926954 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:30.926933 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz"] Nov 02 23:04:31.007029 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:31.006988 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9048ee3-8ca0-4723-9009-f6033427b829-tls-certs\") pod \"premium-w4-637fdb-kserve-79868cf966-dklxz\" (UID: \"a9048ee3-8ca0-4723-9009-f6033427b829\") " pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" Nov 02 23:04:31.007245 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:31.007042 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9048ee3-8ca0-4723-9009-f6033427b829-dshm\") pod \"premium-w4-637fdb-kserve-79868cf966-dklxz\" (UID: \"a9048ee3-8ca0-4723-9009-f6033427b829\") " pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" Nov 02 23:04:31.007245 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:31.007066 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9048ee3-8ca0-4723-9009-f6033427b829-home\") pod \"premium-w4-637fdb-kserve-79868cf966-dklxz\" (UID: \"a9048ee3-8ca0-4723-9009-f6033427b829\") " pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" Nov 02 23:04:31.007347 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:31.007299 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9048ee3-8ca0-4723-9009-f6033427b829-model-cache\") pod \"premium-w4-637fdb-kserve-79868cf966-dklxz\" (UID: \"a9048ee3-8ca0-4723-9009-f6033427b829\") " pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" Nov 02 23:04:31.007401 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:31.007382 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9048ee3-8ca0-4723-9009-f6033427b829-tmp-dir\") pod \"premium-w4-637fdb-kserve-79868cf966-dklxz\" (UID: \"a9048ee3-8ca0-4723-9009-f6033427b829\") " pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" Nov 02 23:04:31.007441 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:31.007423 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj6dj\" (UniqueName: \"kubernetes.io/projected/a9048ee3-8ca0-4723-9009-f6033427b829-kube-api-access-kj6dj\") pod \"premium-w4-637fdb-kserve-79868cf966-dklxz\" (UID: \"a9048ee3-8ca0-4723-9009-f6033427b829\") " pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" Nov 02 23:04:31.108854 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:31.108783 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9048ee3-8ca0-4723-9009-f6033427b829-tls-certs\") pod \"premium-w4-637fdb-kserve-79868cf966-dklxz\" (UID: \"a9048ee3-8ca0-4723-9009-f6033427b829\") " pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" Nov 02 23:04:31.108854 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:31.108815 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9048ee3-8ca0-4723-9009-f6033427b829-dshm\") pod \"premium-w4-637fdb-kserve-79868cf966-dklxz\" (UID: \"a9048ee3-8ca0-4723-9009-f6033427b829\") " pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" Nov 02 23:04:31.108854 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:31.108833 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9048ee3-8ca0-4723-9009-f6033427b829-home\") pod \"premium-w4-637fdb-kserve-79868cf966-dklxz\" (UID: \"a9048ee3-8ca0-4723-9009-f6033427b829\") " pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" Nov 02 23:04:31.109068 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:31.108979 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9048ee3-8ca0-4723-9009-f6033427b829-model-cache\") pod \"premium-w4-637fdb-kserve-79868cf966-dklxz\" (UID: \"a9048ee3-8ca0-4723-9009-f6033427b829\") " pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" Nov 02 23:04:31.109068 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:31.109051 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9048ee3-8ca0-4723-9009-f6033427b829-tmp-dir\") pod \"premium-w4-637fdb-kserve-79868cf966-dklxz\" (UID: \"a9048ee3-8ca0-4723-9009-f6033427b829\") " pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" Nov 02 23:04:31.109269 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:31.109084 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj6dj\" (UniqueName: \"kubernetes.io/projected/a9048ee3-8ca0-4723-9009-f6033427b829-kube-api-access-kj6dj\") pod \"premium-w4-637fdb-kserve-79868cf966-dklxz\" (UID: \"a9048ee3-8ca0-4723-9009-f6033427b829\") " pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" Nov 02 23:04:31.109269 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:31.109255 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9048ee3-8ca0-4723-9009-f6033427b829-home\") pod \"premium-w4-637fdb-kserve-79868cf966-dklxz\" (UID: \"a9048ee3-8ca0-4723-9009-f6033427b829\") " pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" Nov 02 23:04:31.109408 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:31.109388 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9048ee3-8ca0-4723-9009-f6033427b829-model-cache\") pod \"premium-w4-637fdb-kserve-79868cf966-dklxz\" (UID: \"a9048ee3-8ca0-4723-9009-f6033427b829\") " pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" Nov 02 23:04:31.109466 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:31.109425 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9048ee3-8ca0-4723-9009-f6033427b829-tmp-dir\") pod \"premium-w4-637fdb-kserve-79868cf966-dklxz\" (UID: \"a9048ee3-8ca0-4723-9009-f6033427b829\") " pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" Nov 02 23:04:31.111791 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:31.111766 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9048ee3-8ca0-4723-9009-f6033427b829-dshm\") pod \"premium-w4-637fdb-kserve-79868cf966-dklxz\" (UID: \"a9048ee3-8ca0-4723-9009-f6033427b829\") " pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" Nov 02 23:04:31.111986 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:31.111970 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9048ee3-8ca0-4723-9009-f6033427b829-tls-certs\") pod \"premium-w4-637fdb-kserve-79868cf966-dklxz\" (UID: \"a9048ee3-8ca0-4723-9009-f6033427b829\") " pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" Nov 02 23:04:31.169925 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:31.169899 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj6dj\" (UniqueName: \"kubernetes.io/projected/a9048ee3-8ca0-4723-9009-f6033427b829-kube-api-access-kj6dj\") pod \"premium-w4-637fdb-kserve-79868cf966-dklxz\" (UID: \"a9048ee3-8ca0-4723-9009-f6033427b829\") " pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" Nov 02 23:04:31.431920 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:31.431832 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" Nov 02 23:04:31.809569 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:31.809540 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz"] Nov 02 23:04:31.811043 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:04:31.811010 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9048ee3_8ca0_4723_9009_f6033427b829.slice/crio-1041a79ae67fa18c2316bde721751574b07400038c5a26809cac3c62114293c1 WatchSource:0}: Error finding container 1041a79ae67fa18c2316bde721751574b07400038c5a26809cac3c62114293c1: Status 404 returned error can't find the container with id 1041a79ae67fa18c2316bde721751574b07400038c5a26809cac3c62114293c1 Nov 02 23:04:31.962892 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:31.962861 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" event={"ID":"a9048ee3-8ca0-4723-9009-f6033427b829","Type":"ContainerStarted","Data":"44898f25bc176ffdbebb68b06f8174bbe03c58f8fb1a2a25edaee6a93bf61943"} Nov 02 23:04:31.963397 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:31.962901 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" event={"ID":"a9048ee3-8ca0-4723-9009-f6033427b829","Type":"ContainerStarted","Data":"1041a79ae67fa18c2316bde721751574b07400038c5a26809cac3c62114293c1"} Nov 02 23:04:32.079670 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:32.079624 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" podStartSLOduration=2.079610457 podStartE2EDuration="2.079610457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:04:32.077501907 +0000 UTC m=+958.315999590" watchObservedRunningTime="2026-11-02 23:04:32.079610457 +0000 UTC m=+958.318108122" Nov 02 23:04:32.432585 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:32.432483 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" Nov 02 23:04:40.842618 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:40.842576 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd"] Nov 02 23:04:40.843010 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:40.842884 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" podUID="0d966bed-7db8-4d7f-aad4-a9f00d168242" containerName="istio-proxy" containerID="cri-o://7028878b2e5bd4c61623408018f7f1388b31c5e3b2b743beacb534ca46590ec8" gracePeriod=30 Nov 02 23:04:41.432841 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:41.432803 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" Nov 02 23:04:41.445216 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:41.445188 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" Nov 02 23:04:42.041431 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:42.041406 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" Nov 02 23:04:46.073175 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:46.069410 5727 generic.go:356] "Generic (PLEG): container finished" podID="0d966bed-7db8-4d7f-aad4-a9f00d168242" containerID="7028878b2e5bd4c61623408018f7f1388b31c5e3b2b743beacb534ca46590ec8" exitCode=0 Nov 02 23:04:46.073175 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:46.069455 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" event={"ID":"0d966bed-7db8-4d7f-aad4-a9f00d168242","Type":"ContainerDied","Data":"7028878b2e5bd4c61623408018f7f1388b31c5e3b2b743beacb534ca46590ec8"} Nov 02 23:04:46.133374 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:46.133347 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 23:04:46.162554 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:46.162523 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-istio-envoy\") pod \"0d966bed-7db8-4d7f-aad4-a9f00d168242\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " Nov 02 23:04:46.162703 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:46.162564 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d966bed-7db8-4d7f-aad4-a9f00d168242-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d966bed-7db8-4d7f-aad4-a9f00d168242-istiod-ca-cert\") pod \"0d966bed-7db8-4d7f-aad4-a9f00d168242\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " Nov 02 23:04:46.162703 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:46.162588 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-credential-socket\") pod \"0d966bed-7db8-4d7f-aad4-a9f00d168242\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " Nov 02 23:04:46.162819 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:46.162769 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-istio-data\") pod \"0d966bed-7db8-4d7f-aad4-a9f00d168242\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " Nov 02 23:04:46.162873 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:46.162826 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-workload-certs\") pod \"0d966bed-7db8-4d7f-aad4-a9f00d168242\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " Nov 02 23:04:46.162927 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:46.162865 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-credential-socket" pod "0d966bed-7db8-4d7f-aad4-a9f00d168242" (UID: "0d966bed-7db8-4d7f-aad4-a9f00d168242"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:04:46.162994 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:46.162977 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d966bed-7db8-4d7f-aad4-a9f00d168242-kube-api-access-zcqkr\" (UniqueName: \"kubernetes.io/projected/0d966bed-7db8-4d7f-aad4-a9f00d168242-kube-api-access-zcqkr\") pod \"0d966bed-7db8-4d7f-aad4-a9f00d168242\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " Nov 02 23:04:46.163082 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:46.163061 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-workload-socket\") pod \"0d966bed-7db8-4d7f-aad4-a9f00d168242\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " Nov 02 23:04:46.163249 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:46.163148 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d966bed-7db8-4d7f-aad4-a9f00d168242-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d966bed-7db8-4d7f-aad4-a9f00d168242-istio-podinfo\") pod \"0d966bed-7db8-4d7f-aad4-a9f00d168242\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " Nov 02 23:04:46.163249 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:46.163205 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d966bed-7db8-4d7f-aad4-a9f00d168242-istio-token\" (UniqueName: \"kubernetes.io/projected/0d966bed-7db8-4d7f-aad4-a9f00d168242-istio-token\") pod \"0d966bed-7db8-4d7f-aad4-a9f00d168242\" (UID: \"0d966bed-7db8-4d7f-aad4-a9f00d168242\") " Nov 02 23:04:46.163370 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:46.163325 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-workload-socket" pod "0d966bed-7db8-4d7f-aad4-a9f00d168242" (UID: "0d966bed-7db8-4d7f-aad4-a9f00d168242"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:04:46.163370 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:46.162921 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d966bed-7db8-4d7f-aad4-a9f00d168242-istiod-ca-cert" pod "0d966bed-7db8-4d7f-aad4-a9f00d168242" (UID: "0d966bed-7db8-4d7f-aad4-a9f00d168242"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:04:46.163370 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:46.163022 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-istio-data" pod "0d966bed-7db8-4d7f-aad4-a9f00d168242" (UID: "0d966bed-7db8-4d7f-aad4-a9f00d168242"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:04:46.163565 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:46.163094 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-workload-certs" pod "0d966bed-7db8-4d7f-aad4-a9f00d168242" (UID: "0d966bed-7db8-4d7f-aad4-a9f00d168242"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:04:46.163806 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:46.163784 5727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-workload-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:04:46.163957 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:46.163935 5727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d966bed-7db8-4d7f-aad4-a9f00d168242-istiod-ca-cert\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:04:46.164048 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:46.163962 5727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-credential-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:04:46.164048 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:46.163977 5727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-istio-data\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:04:46.164048 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:46.163991 5727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-workload-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:04:46.165176 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:46.165154 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-istio-envoy" pod "0d966bed-7db8-4d7f-aad4-a9f00d168242" (UID: "0d966bed-7db8-4d7f-aad4-a9f00d168242"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:04:46.165482 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:46.165459 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d966bed-7db8-4d7f-aad4-a9f00d168242-kube-api-access-zcqkr" pod "0d966bed-7db8-4d7f-aad4-a9f00d168242" (UID: "0d966bed-7db8-4d7f-aad4-a9f00d168242"). InnerVolumeSpecName "kube-api-access-zcqkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:04:46.165676 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:46.165657 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d966bed-7db8-4d7f-aad4-a9f00d168242-istio-token" pod "0d966bed-7db8-4d7f-aad4-a9f00d168242" (UID: "0d966bed-7db8-4d7f-aad4-a9f00d168242"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:04:46.165906 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:46.165880 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d966bed-7db8-4d7f-aad4-a9f00d168242-istio-podinfo" pod "0d966bed-7db8-4d7f-aad4-a9f00d168242" (UID: "0d966bed-7db8-4d7f-aad4-a9f00d168242"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:04:46.265310 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:46.265275 5727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d966bed-7db8-4d7f-aad4-a9f00d168242-istio-podinfo\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:04:46.265310 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:46.265307 5727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d966bed-7db8-4d7f-aad4-a9f00d168242-istio-token\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:04:46.265488 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:46.265321 5727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d966bed-7db8-4d7f-aad4-a9f00d168242-istio-envoy\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:04:46.265488 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:46.265335 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcqkr\" (UniqueName: \"kubernetes.io/projected/0d966bed-7db8-4d7f-aad4-a9f00d168242-kube-api-access-zcqkr\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:04:47.078647 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:47.078618 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" Nov 02 23:04:47.079089 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:47.078654 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd" event={"ID":"0d966bed-7db8-4d7f-aad4-a9f00d168242","Type":"ContainerDied","Data":"4dca1d914ea782047cac222b9a19ff5b1338e322a5bd0fc27574359e1bc4f77d"} Nov 02 23:04:47.079089 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:47.078724 5727 scope.go:122] "RemoveContainer" containerID="7028878b2e5bd4c61623408018f7f1388b31c5e3b2b743beacb534ca46590ec8" Nov 02 23:04:47.295393 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:47.295362 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd"] Nov 02 23:04:47.328221 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:47.328190 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7dzqd"] Nov 02 23:04:47.884096 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:47.884064 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d966bed-7db8-4d7f-aad4-a9f00d168242" path="/var/lib/kubelet/pods/0d966bed-7db8-4d7f-aad4-a9f00d168242/volumes" Nov 02 23:04:55.250835 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:55.250799 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5"] Nov 02 23:04:55.251588 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:55.251566 5727 cpu_manager.go:414] "containerMap: removing container" podUID="0d966bed-7db8-4d7f-aad4-a9f00d168242" containerName="istio-proxy" Nov 02 23:04:55.251699 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:55.251589 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d966bed-7db8-4d7f-aad4-a9f00d168242" containerName="istio-proxy" Nov 02 23:04:55.251759 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:55.251737 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d966bed-7db8-4d7f-aad4-a9f00d168242" containerName="istio-proxy" Nov 02 23:04:55.255095 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:55.255075 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" Nov 02 23:04:55.358753 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:55.358710 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b42983\"/\"openshift-service-ca.crt\"" Nov 02 23:04:55.358753 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:55.358729 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b42983\"/\"premium-w1-b42983-kserve-self-signed-certs\"" Nov 02 23:04:55.359002 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:55.358888 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b42983\"/\"kube-root-ca.crt\"" Nov 02 23:04:55.359002 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:55.358932 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b42983\"/\"default-dockercfg-qt2p2\"" Nov 02 23:04:55.361521 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:55.361497 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5"] Nov 02 23:04:55.369425 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:55.369403 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-tls-certs\") pod \"premium-w1-b42983-kserve-865c76d74c-clnj5\" (UID: \"f83c8651-5db1-4e80-b9fb-ec5b793e7e94\") " pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" Nov 02 23:04:55.369542 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:55.369463 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnvxc\" (UniqueName: \"kubernetes.io/projected/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-kube-api-access-fnvxc\") pod \"premium-w1-b42983-kserve-865c76d74c-clnj5\" (UID: \"f83c8651-5db1-4e80-b9fb-ec5b793e7e94\") " pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" Nov 02 23:04:55.369542 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:55.369521 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-dshm\") pod \"premium-w1-b42983-kserve-865c76d74c-clnj5\" (UID: \"f83c8651-5db1-4e80-b9fb-ec5b793e7e94\") " pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" Nov 02 23:04:55.369617 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:55.369579 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-model-cache\") pod \"premium-w1-b42983-kserve-865c76d74c-clnj5\" (UID: \"f83c8651-5db1-4e80-b9fb-ec5b793e7e94\") " pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" Nov 02 23:04:55.369617 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:55.369606 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-tmp-dir\") pod \"premium-w1-b42983-kserve-865c76d74c-clnj5\" (UID: \"f83c8651-5db1-4e80-b9fb-ec5b793e7e94\") " pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" Nov 02 23:04:55.369723 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:55.369707 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-home\") pod \"premium-w1-b42983-kserve-865c76d74c-clnj5\" (UID: \"f83c8651-5db1-4e80-b9fb-ec5b793e7e94\") " pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" Nov 02 23:04:55.470889 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:55.470853 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-model-cache\") pod \"premium-w1-b42983-kserve-865c76d74c-clnj5\" (UID: \"f83c8651-5db1-4e80-b9fb-ec5b793e7e94\") " pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" Nov 02 23:04:55.471079 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:55.470896 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-tmp-dir\") pod \"premium-w1-b42983-kserve-865c76d74c-clnj5\" (UID: \"f83c8651-5db1-4e80-b9fb-ec5b793e7e94\") " pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" Nov 02 23:04:55.471079 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:55.470941 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-home\") pod \"premium-w1-b42983-kserve-865c76d74c-clnj5\" (UID: \"f83c8651-5db1-4e80-b9fb-ec5b793e7e94\") " pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" Nov 02 23:04:55.471079 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:55.471002 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-tls-certs\") pod \"premium-w1-b42983-kserve-865c76d74c-clnj5\" (UID: \"f83c8651-5db1-4e80-b9fb-ec5b793e7e94\") " pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" Nov 02 23:04:55.471079 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:55.471072 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnvxc\" (UniqueName: \"kubernetes.io/projected/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-kube-api-access-fnvxc\") pod \"premium-w1-b42983-kserve-865c76d74c-clnj5\" (UID: \"f83c8651-5db1-4e80-b9fb-ec5b793e7e94\") " pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" Nov 02 23:04:55.471323 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:55.471098 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-dshm\") pod \"premium-w1-b42983-kserve-865c76d74c-clnj5\" (UID: \"f83c8651-5db1-4e80-b9fb-ec5b793e7e94\") " pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" Nov 02 23:04:55.471402 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:55.471382 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-model-cache\") pod \"premium-w1-b42983-kserve-865c76d74c-clnj5\" (UID: \"f83c8651-5db1-4e80-b9fb-ec5b793e7e94\") " pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" Nov 02 23:04:55.471462 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:55.471413 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-tmp-dir\") pod \"premium-w1-b42983-kserve-865c76d74c-clnj5\" (UID: \"f83c8651-5db1-4e80-b9fb-ec5b793e7e94\") " pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" Nov 02 23:04:55.471520 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:55.471456 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-home\") pod \"premium-w1-b42983-kserve-865c76d74c-clnj5\" (UID: \"f83c8651-5db1-4e80-b9fb-ec5b793e7e94\") " pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" Nov 02 23:04:55.473603 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:55.473582 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-dshm\") pod \"premium-w1-b42983-kserve-865c76d74c-clnj5\" (UID: \"f83c8651-5db1-4e80-b9fb-ec5b793e7e94\") " pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" Nov 02 23:04:55.473794 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:55.473775 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-tls-certs\") pod \"premium-w1-b42983-kserve-865c76d74c-clnj5\" (UID: \"f83c8651-5db1-4e80-b9fb-ec5b793e7e94\") " pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" Nov 02 23:04:55.531474 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:55.531402 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnvxc\" (UniqueName: \"kubernetes.io/projected/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-kube-api-access-fnvxc\") pod \"premium-w1-b42983-kserve-865c76d74c-clnj5\" (UID: \"f83c8651-5db1-4e80-b9fb-ec5b793e7e94\") " pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" Nov 02 23:04:55.579104 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:55.579077 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" Nov 02 23:04:55.967667 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:55.967638 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5"] Nov 02 23:04:55.969170 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:04:55.969110 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf83c8651_5db1_4e80_b9fb_ec5b793e7e94.slice/crio-ba123b3cd1f77756b3ab8074d90a929a3b67369cceaec80003672e5343474fb8 WatchSource:0}: Error finding container ba123b3cd1f77756b3ab8074d90a929a3b67369cceaec80003672e5343474fb8: Status 404 returned error can't find the container with id ba123b3cd1f77756b3ab8074d90a929a3b67369cceaec80003672e5343474fb8 Nov 02 23:04:56.139394 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:56.139362 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" event={"ID":"f83c8651-5db1-4e80-b9fb-ec5b793e7e94","Type":"ContainerStarted","Data":"190e9d3ee7b477fdb9ac882f06e6b972dce843fcc7bdcfb0048fceb1ff225b72"} Nov 02 23:04:56.139577 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:56.139408 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" event={"ID":"f83c8651-5db1-4e80-b9fb-ec5b793e7e94","Type":"ContainerStarted","Data":"ba123b3cd1f77756b3ab8074d90a929a3b67369cceaec80003672e5343474fb8"} Nov 02 23:04:56.256489 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:56.256385 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" podStartSLOduration=1.256372039 podStartE2EDuration="1.256372039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:04:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:04:56.255936961 +0000 UTC m=+982.494434644" watchObservedRunningTime="2026-11-02 23:04:56.256372039 +0000 UTC m=+982.494869700" Nov 02 23:04:56.579453 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:04:56.579424 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" Nov 02 23:04:57.844569 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:04:57.844535 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:05:05.580213 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:05.580172 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" Nov 02 23:05:05.592496 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:05.592473 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" Nov 02 23:05:06.222324 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:06.222291 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" Nov 02 23:05:27.843483 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:05:27.843444 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:05:46.642435 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:46.642402 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v"] Nov 02 23:05:46.648055 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:46.648035 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:46.700242 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:46.700216 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-08c2c6-openshift-default-dockercfg-qcc5b\"" Nov 02 23:05:46.753181 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:46.753118 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v"] Nov 02 23:05:46.759177 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:46.759111 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:46.759310 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:46.759283 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:46.759353 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:46.759338 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/575d202b-f514-4312-8dc7-dfad51c6ca19-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:46.759462 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:46.759440 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qddss\" (UniqueName: \"kubernetes.io/projected/575d202b-f514-4312-8dc7-dfad51c6ca19-kube-api-access-qddss\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:46.759523 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:46.759506 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/575d202b-f514-4312-8dc7-dfad51c6ca19-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:46.759634 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:46.759610 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/575d202b-f514-4312-8dc7-dfad51c6ca19-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:46.759683 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:46.759654 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:46.759774 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:46.759756 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:46.759838 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:46.759798 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:46.861173 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:46.861105 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/575d202b-f514-4312-8dc7-dfad51c6ca19-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:46.861370 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:46.861191 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qddss\" (UniqueName: \"kubernetes.io/projected/575d202b-f514-4312-8dc7-dfad51c6ca19-kube-api-access-qddss\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:46.861370 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:46.861235 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/575d202b-f514-4312-8dc7-dfad51c6ca19-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:46.861370 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:46.861280 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/575d202b-f514-4312-8dc7-dfad51c6ca19-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:46.861370 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:46.861307 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:46.861370 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:46.861349 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:46.861635 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:46.861384 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:46.861635 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:46.861473 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:46.861635 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:46.861580 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:46.861789 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:46.861724 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:46.861848 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:46.861793 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:46.861904 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:46.861884 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:46.861998 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:46.861971 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/575d202b-f514-4312-8dc7-dfad51c6ca19-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:46.862113 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:46.861971 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:46.863756 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:46.863736 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/575d202b-f514-4312-8dc7-dfad51c6ca19-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:46.863836 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:46.863809 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:46.921888 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:46.921814 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/575d202b-f514-4312-8dc7-dfad51c6ca19-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:46.922018 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:46.921885 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qddss\" (UniqueName: \"kubernetes.io/projected/575d202b-f514-4312-8dc7-dfad51c6ca19-kube-api-access-qddss\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:46.961429 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:46.961383 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:47.393367 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:47.393344 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v"] Nov 02 23:05:47.394779 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:05:47.394745 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod575d202b_f514_4312_8dc7_dfad51c6ca19.slice/crio-3bd096dd8f0cacaa1236efbe4eb2f6103a0d37bdb672033baf1671ea9cff0bc1 WatchSource:0}: Error finding container 3bd096dd8f0cacaa1236efbe4eb2f6103a0d37bdb672033baf1671ea9cff0bc1: Status 404 returned error can't find the container with id 3bd096dd8f0cacaa1236efbe4eb2f6103a0d37bdb672033baf1671ea9cff0bc1 Nov 02 23:05:47.397144 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:47.397099 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:05:47.397236 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:47.397188 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:05:47.397236 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:47.397220 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:05:47.568910 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:47.568878 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" event={"ID":"575d202b-f514-4312-8dc7-dfad51c6ca19","Type":"ContainerStarted","Data":"983d66f724bc1f6dac51c13a78a6ac56816866652d92394fdee53a768d587f1b"} Nov 02 23:05:47.569091 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:47.568936 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" event={"ID":"575d202b-f514-4312-8dc7-dfad51c6ca19","Type":"ContainerStarted","Data":"3bd096dd8f0cacaa1236efbe4eb2f6103a0d37bdb672033baf1671ea9cff0bc1"} Nov 02 23:05:47.693348 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:47.693225 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" podStartSLOduration=1.693207122 podStartE2EDuration="1.693207122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:05:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:05:47.692533843 +0000 UTC m=+1033.931031528" watchObservedRunningTime="2026-11-02 23:05:47.693207122 +0000 UTC m=+1033.931704785" Nov 02 23:05:47.975216 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:47.961470 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:48.976680 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:48.966764 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 02 23:05:48.976680 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:48.966826 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" podUID="575d202b-f514-4312-8dc7-dfad51c6ca19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 02 23:05:49.973156 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:49.972545 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 02 23:05:49.973156 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:49.972613 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" podUID="575d202b-f514-4312-8dc7-dfad51c6ca19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 02 23:05:50.965451 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:50.965417 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 02 23:05:50.966025 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:50.965995 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" podUID="575d202b-f514-4312-8dc7-dfad51c6ca19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 02 23:05:51.963511 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:51.963461 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 02 23:05:51.963789 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:51.963762 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" podUID="575d202b-f514-4312-8dc7-dfad51c6ca19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 02 23:05:52.963559 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:52.963518 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 02 23:05:52.964272 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:52.964240 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" podUID="575d202b-f514-4312-8dc7-dfad51c6ca19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 02 23:05:54.085108 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:54.085073 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:54.085968 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:54.085942 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:54.086719 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:54.086701 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:05:56.191826 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.191790 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df"] Nov 02 23:05:56.195892 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.195875 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:05:56.264058 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.264024 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-7g7df\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:05:56.264252 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.264105 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-7g7df\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:05:56.264321 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.264246 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ee46cd-bfd4-439b-a15c-394e7d10b915-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-7g7df\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:05:56.264321 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.264281 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-7g7df\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:05:56.264321 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.264310 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ee46cd-bfd4-439b-a15c-394e7d10b915-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-7g7df\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:05:56.264502 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.264474 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-7g7df\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:05:56.264557 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.264538 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxdx8\" (UniqueName: \"kubernetes.io/projected/65ee46cd-bfd4-439b-a15c-394e7d10b915-kube-api-access-nxdx8\") pod \"maas-default-gateway-openshift-default-694db55c87-7g7df\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:05:56.264696 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.264676 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-7g7df\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:05:56.264755 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.264726 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65ee46cd-bfd4-439b-a15c-394e7d10b915-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-7g7df\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:05:56.371170 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.367733 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-7g7df\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:05:56.371170 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.367838 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-7g7df\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:05:56.371170 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.367895 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ee46cd-bfd4-439b-a15c-394e7d10b915-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-7g7df\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:05:56.371170 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.367927 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-7g7df\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:05:56.371170 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.367957 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ee46cd-bfd4-439b-a15c-394e7d10b915-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-7g7df\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:05:56.371170 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.368008 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-7g7df\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:05:56.371170 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.368054 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxdx8\" (UniqueName: \"kubernetes.io/projected/65ee46cd-bfd4-439b-a15c-394e7d10b915-kube-api-access-nxdx8\") pod \"maas-default-gateway-openshift-default-694db55c87-7g7df\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:05:56.371170 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.368149 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-7g7df\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:05:56.371170 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.368195 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65ee46cd-bfd4-439b-a15c-394e7d10b915-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-7g7df\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:05:56.371170 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.368850 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-7g7df\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:05:56.371170 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.369148 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-7g7df\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:05:56.371170 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.369549 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-7g7df\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:05:56.371170 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.370212 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ee46cd-bfd4-439b-a15c-394e7d10b915-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-7g7df\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:05:56.371170 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.370579 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-7g7df\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:05:56.372345 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.372318 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-7g7df\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:05:56.378289 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.378261 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ee46cd-bfd4-439b-a15c-394e7d10b915-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-7g7df\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:05:56.428773 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.428743 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65ee46cd-bfd4-439b-a15c-394e7d10b915-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-7g7df\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:05:56.428924 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.428857 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxdx8\" (UniqueName: \"kubernetes.io/projected/65ee46cd-bfd4-439b-a15c-394e7d10b915-kube-api-access-nxdx8\") pod \"maas-default-gateway-openshift-default-694db55c87-7g7df\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:05:56.443006 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.442937 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df"] Nov 02 23:05:56.513840 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.513812 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:05:56.954638 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.954614 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df"] Nov 02 23:05:56.957784 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:05:56.957757 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ee46cd_bfd4_439b_a15c_394e7d10b915.slice/crio-30e5352269723ba7607109279a7290b5b9dcd09ee71387966b41cd9e8f408fc1 WatchSource:0}: Error finding container 30e5352269723ba7607109279a7290b5b9dcd09ee71387966b41cd9e8f408fc1: Status 404 returned error can't find the container with id 30e5352269723ba7607109279a7290b5b9dcd09ee71387966b41cd9e8f408fc1 Nov 02 23:05:56.960674 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.960643 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:05:56.960767 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.960701 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:05:56.960767 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:56.960731 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:05:57.684977 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:57.684943 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" event={"ID":"65ee46cd-bfd4-439b-a15c-394e7d10b915","Type":"ContainerStarted","Data":"b9a106765c32a5935e454d98227962009e2693dd6e666a0557e50e196c12c2b6"} Nov 02 23:05:57.685442 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:57.684999 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" event={"ID":"65ee46cd-bfd4-439b-a15c-394e7d10b915","Type":"ContainerStarted","Data":"30e5352269723ba7607109279a7290b5b9dcd09ee71387966b41cd9e8f408fc1"} Nov 02 23:05:57.849018 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:05:57.848980 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:05:58.518064 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:58.517291 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:05:58.519978 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:58.519948 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-7g7df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 02 23:05:58.520109 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:58.520010 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" podUID="65ee46cd-bfd4-439b-a15c-394e7d10b915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 02 23:05:59.514675 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:59.514642 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-7g7df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 02 23:05:59.515172 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:05:59.514701 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" podUID="65ee46cd-bfd4-439b-a15c-394e7d10b915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 02 23:06:00.514210 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:00.514174 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-7g7df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 02 23:06:00.514387 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:00.514245 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" podUID="65ee46cd-bfd4-439b-a15c-394e7d10b915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 02 23:06:01.514087 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:01.514053 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-7g7df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 02 23:06:01.514499 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:01.514106 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" podUID="65ee46cd-bfd4-439b-a15c-394e7d10b915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 02 23:06:02.023338 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:02.023286 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" podStartSLOduration=6.02327296 podStartE2EDuration="6.02327296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:05:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:05:57.867332796 +0000 UTC m=+1044.105830458" watchObservedRunningTime="2026-11-02 23:06:02.02327296 +0000 UTC m=+1048.261770622" Nov 02 23:06:02.023677 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:02.023663 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7"] Nov 02 23:06:02.027782 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:02.027766 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" Nov 02 23:06:02.130527 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:02.130492 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-08c2c6\"/\"kube-root-ca.crt\"" Nov 02 23:06:02.130714 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:02.130558 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-08c2c6\"/\"openshift-service-ca.crt\"" Nov 02 23:06:02.130714 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:02.130598 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-08c2c6\"/\"e2e-conflict-a-4cba9672-kserve-self-signed-certs\"" Nov 02 23:06:02.130714 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:02.130607 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-08c2c6\"/\"default-dockercfg-zg2jb\"" Nov 02 23:06:02.133496 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:02.133476 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7"] Nov 02 23:06:02.148252 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:02.148225 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b724650-f2c7-4a4b-a077-edf537b42392-dshm\") pod \"e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7\" (UID: \"4b724650-f2c7-4a4b-a077-edf537b42392\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" Nov 02 23:06:02.148365 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:02.148259 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b724650-f2c7-4a4b-a077-edf537b42392-tls-certs\") pod \"e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7\" (UID: \"4b724650-f2c7-4a4b-a077-edf537b42392\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" Nov 02 23:06:02.148365 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:02.148296 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b724650-f2c7-4a4b-a077-edf537b42392-model-cache\") pod \"e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7\" (UID: \"4b724650-f2c7-4a4b-a077-edf537b42392\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" Nov 02 23:06:02.148478 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:02.148454 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr9g6\" (UniqueName: \"kubernetes.io/projected/4b724650-f2c7-4a4b-a077-edf537b42392-kube-api-access-mr9g6\") pod \"e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7\" (UID: \"4b724650-f2c7-4a4b-a077-edf537b42392\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" Nov 02 23:06:02.148551 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:02.148536 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b724650-f2c7-4a4b-a077-edf537b42392-tmp-dir\") pod \"e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7\" (UID: \"4b724650-f2c7-4a4b-a077-edf537b42392\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" Nov 02 23:06:02.148615 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:02.148588 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b724650-f2c7-4a4b-a077-edf537b42392-home\") pod \"e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7\" (UID: \"4b724650-f2c7-4a4b-a077-edf537b42392\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" Nov 02 23:06:02.250067 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:02.250036 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b724650-f2c7-4a4b-a077-edf537b42392-dshm\") pod \"e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7\" (UID: \"4b724650-f2c7-4a4b-a077-edf537b42392\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" Nov 02 23:06:02.250293 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:02.250073 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b724650-f2c7-4a4b-a077-edf537b42392-tls-certs\") pod \"e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7\" (UID: \"4b724650-f2c7-4a4b-a077-edf537b42392\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" Nov 02 23:06:02.250293 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:02.250207 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b724650-f2c7-4a4b-a077-edf537b42392-model-cache\") pod \"e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7\" (UID: \"4b724650-f2c7-4a4b-a077-edf537b42392\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" Nov 02 23:06:02.250419 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:02.250304 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr9g6\" (UniqueName: \"kubernetes.io/projected/4b724650-f2c7-4a4b-a077-edf537b42392-kube-api-access-mr9g6\") pod \"e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7\" (UID: \"4b724650-f2c7-4a4b-a077-edf537b42392\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" Nov 02 23:06:02.250419 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:02.250377 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b724650-f2c7-4a4b-a077-edf537b42392-tmp-dir\") pod \"e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7\" (UID: \"4b724650-f2c7-4a4b-a077-edf537b42392\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" Nov 02 23:06:02.250516 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:02.250429 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b724650-f2c7-4a4b-a077-edf537b42392-home\") pod \"e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7\" (UID: \"4b724650-f2c7-4a4b-a077-edf537b42392\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" Nov 02 23:06:02.255154 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:02.250864 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b724650-f2c7-4a4b-a077-edf537b42392-home\") pod \"e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7\" (UID: \"4b724650-f2c7-4a4b-a077-edf537b42392\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" Nov 02 23:06:02.255154 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:02.251101 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b724650-f2c7-4a4b-a077-edf537b42392-model-cache\") pod \"e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7\" (UID: \"4b724650-f2c7-4a4b-a077-edf537b42392\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" Nov 02 23:06:02.255154 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:02.251671 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b724650-f2c7-4a4b-a077-edf537b42392-tmp-dir\") pod \"e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7\" (UID: \"4b724650-f2c7-4a4b-a077-edf537b42392\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" Nov 02 23:06:02.281296 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:02.280886 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b724650-f2c7-4a4b-a077-edf537b42392-dshm\") pod \"e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7\" (UID: \"4b724650-f2c7-4a4b-a077-edf537b42392\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" Nov 02 23:06:02.281764 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:02.281743 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b724650-f2c7-4a4b-a077-edf537b42392-tls-certs\") pod \"e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7\" (UID: \"4b724650-f2c7-4a4b-a077-edf537b42392\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" Nov 02 23:06:02.312050 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:02.312025 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr9g6\" (UniqueName: \"kubernetes.io/projected/4b724650-f2c7-4a4b-a077-edf537b42392-kube-api-access-mr9g6\") pod \"e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7\" (UID: \"4b724650-f2c7-4a4b-a077-edf537b42392\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" Nov 02 23:06:02.342194 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:02.342167 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" Nov 02 23:06:02.514898 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:02.514452 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-7g7df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 02 23:06:02.514898 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:02.514522 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" podUID="65ee46cd-bfd4-439b-a15c-394e7d10b915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 02 23:06:02.802473 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:02.801147 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7"] Nov 02 23:06:03.266518 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.266481 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml"] Nov 02 23:06:03.270799 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.270775 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:03.322980 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.322943 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-637fdb-openshift-default-dockercfg-brzsm\"" Nov 02 23:06:03.331935 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.331907 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml"] Nov 02 23:06:03.394218 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.394097 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:03.394218 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.394184 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f88cb2c-1bec-4b58-a525-326c52c23931-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:03.394480 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.394285 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:03.394480 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.394357 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj8fq\" (UniqueName: \"kubernetes.io/projected/6f88cb2c-1bec-4b58-a525-326c52c23931-kube-api-access-zj8fq\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:03.394480 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.394417 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f88cb2c-1bec-4b58-a525-326c52c23931-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:03.394480 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.394450 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:03.394700 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.394482 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:03.394700 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.394577 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f88cb2c-1bec-4b58-a525-326c52c23931-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:03.394700 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.394607 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:03.496110 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.496072 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f88cb2c-1bec-4b58-a525-326c52c23931-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:03.496315 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.496119 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:03.496387 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.496318 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:03.496446 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.496429 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f88cb2c-1bec-4b58-a525-326c52c23931-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:03.496503 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.496461 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:03.496556 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.496521 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:03.496556 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.496549 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f88cb2c-1bec-4b58-a525-326c52c23931-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:03.496765 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.496632 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:03.496765 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.496694 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj8fq\" (UniqueName: \"kubernetes.io/projected/6f88cb2c-1bec-4b58-a525-326c52c23931-kube-api-access-zj8fq\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:03.497069 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.496908 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:03.497269 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.497237 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:03.497440 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.497414 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:03.497524 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.497506 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:03.497590 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.497557 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f88cb2c-1bec-4b58-a525-326c52c23931-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:03.498831 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.498807 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:03.500099 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.500077 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f88cb2c-1bec-4b58-a525-326c52c23931-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:03.514584 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.514559 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-7g7df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 02 23:06:03.514672 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.514625 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" podUID="65ee46cd-bfd4-439b-a15c-394e7d10b915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 02 23:06:03.557427 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.557363 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f88cb2c-1bec-4b58-a525-326c52c23931-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:03.557947 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.557923 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj8fq\" (UniqueName: \"kubernetes.io/projected/6f88cb2c-1bec-4b58-a525-326c52c23931-kube-api-access-zj8fq\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:03.615466 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.615437 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:03.751177 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.751113 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" event={"ID":"4b724650-f2c7-4a4b-a077-edf537b42392","Type":"ContainerStarted","Data":"4c22d10d5eacf39b3557d51331917bc81d2f4521f0748b1cc3bf9ddb5dfdba72"} Nov 02 23:06:03.751358 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.751194 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" event={"ID":"4b724650-f2c7-4a4b-a077-edf537b42392","Type":"ContainerStarted","Data":"2785dc1e11fda60336489f3639900227b2e84da3e3e4983349034c5027b327e8"} Nov 02 23:06:03.871746 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:03.871677 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" podStartSLOduration=2.871658118 podStartE2EDuration="2.871658118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:06:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:06:03.866025873 +0000 UTC m=+1050.104523558" watchObservedRunningTime="2026-11-02 23:06:03.871658118 +0000 UTC m=+1050.110155781" Nov 02 23:06:04.070453 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:04.070421 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml"] Nov 02 23:06:04.071888 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:06:04.071831 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f88cb2c_1bec_4b58_a525_326c52c23931.slice/crio-47ebfef07de5c1bae3c5808ac6ab06f231498bb00eb62af593c88c33aad013cf WatchSource:0}: Error finding container 47ebfef07de5c1bae3c5808ac6ab06f231498bb00eb62af593c88c33aad013cf: Status 404 returned error can't find the container with id 47ebfef07de5c1bae3c5808ac6ab06f231498bb00eb62af593c88c33aad013cf Nov 02 23:06:04.074590 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:04.074559 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:06:04.074670 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:04.074623 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:06:04.074670 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:04.074651 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:06:04.343183 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:04.343144 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" Nov 02 23:06:04.515164 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:04.515054 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-7g7df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 02 23:06:04.515164 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:04.515110 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" podUID="65ee46cd-bfd4-439b-a15c-394e7d10b915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 02 23:06:04.796688 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:04.796587 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" event={"ID":"6f88cb2c-1bec-4b58-a525-326c52c23931","Type":"ContainerStarted","Data":"cac15f0d77fd386685429e35235a9e9fbf397d066b02006a42c30b02cb552102"} Nov 02 23:06:04.796688 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:04.796648 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" event={"ID":"6f88cb2c-1bec-4b58-a525-326c52c23931","Type":"ContainerStarted","Data":"47ebfef07de5c1bae3c5808ac6ab06f231498bb00eb62af593c88c33aad013cf"} Nov 02 23:06:04.961157 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:04.949862 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" podStartSLOduration=1.949842117 podStartE2EDuration="1.949842117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:06:04.947035313 +0000 UTC m=+1051.185532976" watchObservedRunningTime="2026-11-02 23:06:04.949842117 +0000 UTC m=+1051.188339780" Nov 02 23:06:05.616311 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:05.616277 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:05.619116 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:05.619084 5727 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 02 23:06:05.619301 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:05.619171 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" podUID="6f88cb2c-1bec-4b58-a525-326c52c23931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 02 23:06:05.921557 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:05.921460 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:06:05.922062 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:05.921907 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:06:05.923049 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:05.922990 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:06:06.616565 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:06.616520 5727 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 02 23:06:06.616776 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:06.616576 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" podUID="6f88cb2c-1bec-4b58-a525-326c52c23931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 02 23:06:07.627153 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:07.619614 5727 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 02 23:06:07.627153 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:07.619694 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" podUID="6f88cb2c-1bec-4b58-a525-326c52c23931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 02 23:06:08.620160 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:08.619356 5727 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 02 23:06:08.620160 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:08.619419 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" podUID="6f88cb2c-1bec-4b58-a525-326c52c23931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 02 23:06:09.616548 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:09.616498 5727 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 02 23:06:09.617061 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:09.616562 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" podUID="6f88cb2c-1bec-4b58-a525-326c52c23931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 02 23:06:10.616034 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:10.616000 5727 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 02 23:06:10.616205 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:10.616058 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" podUID="6f88cb2c-1bec-4b58-a525-326c52c23931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 02 23:06:11.153791 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.153763 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw"] Nov 02 23:06:11.158138 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.158112 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:11.260662 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.260623 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-gm8cw\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:11.260863 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.260680 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-gm8cw\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:11.260863 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.260706 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-gm8cw\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:11.260863 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.260820 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-gm8cw\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:11.261025 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.260886 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-gm8cw\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:11.261025 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.260997 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-gm8cw\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:11.261121 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.261029 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-gm8cw\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:11.261121 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.261070 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-gm8cw\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:11.261121 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.261088 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx74d\" (UniqueName: \"kubernetes.io/projected/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-kube-api-access-fx74d\") pod \"maas-default-gateway-openshift-default-694db55c87-gm8cw\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:11.262329 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.262299 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw"] Nov 02 23:06:11.362759 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.362731 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-gm8cw\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:11.362936 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.362771 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-gm8cw\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:11.363012 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.362949 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-gm8cw\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:11.363012 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.362970 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx74d\" (UniqueName: \"kubernetes.io/projected/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-kube-api-access-fx74d\") pod \"maas-default-gateway-openshift-default-694db55c87-gm8cw\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:11.363158 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.363013 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-gm8cw\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:11.363158 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.363041 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-gm8cw\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:11.363158 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.363064 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-gm8cw\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:11.363158 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.363096 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-gm8cw\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:11.363158 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.363114 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-gm8cw\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:11.363457 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.363192 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-gm8cw\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:11.363457 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.363390 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-gm8cw\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:11.363457 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.363409 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-gm8cw\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:11.363687 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.363665 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-gm8cw\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:11.363751 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.363736 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-gm8cw\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:11.365613 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.365593 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-gm8cw\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:11.365703 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.365599 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-gm8cw\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:11.423710 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.423647 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-gm8cw\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:11.423935 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.423912 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx74d\" (UniqueName: \"kubernetes.io/projected/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-kube-api-access-fx74d\") pod \"maas-default-gateway-openshift-default-694db55c87-gm8cw\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:11.472735 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.472707 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:11.619995 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.619967 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:11.620453 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.620433 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:11.621086 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.621066 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:06:11.908286 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.908257 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw"] Nov 02 23:06:11.909722 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:06:11.909699 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cb4d651_3373_4449_8f2f_0cc8f748b1b3.slice/crio-7e47130369c970d5cd76d2851e0832c059ec3b2e40bd5dedddc3d484b5597ec2 WatchSource:0}: Error finding container 7e47130369c970d5cd76d2851e0832c059ec3b2e40bd5dedddc3d484b5597ec2: Status 404 returned error can't find the container with id 7e47130369c970d5cd76d2851e0832c059ec3b2e40bd5dedddc3d484b5597ec2 Nov 02 23:06:11.911933 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.911907 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:06:11.912008 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.911966 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:06:11.912008 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:11.911994 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:06:12.343104 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:12.343071 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" Nov 02 23:06:12.356830 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:12.356803 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" Nov 02 23:06:12.900478 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:12.900445 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" event={"ID":"4cb4d651-3373-4449-8f2f-0cc8f748b1b3","Type":"ContainerStarted","Data":"3cfc6d5d277674e2c4e964b7b33db50bffe4f2b9460b7b4dedae4c1a52bf2940"} Nov 02 23:06:12.900691 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:12.900501 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" event={"ID":"4cb4d651-3373-4449-8f2f-0cc8f748b1b3","Type":"ContainerStarted","Data":"7e47130369c970d5cd76d2851e0832c059ec3b2e40bd5dedddc3d484b5597ec2"} Nov 02 23:06:12.912144 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:12.912098 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" Nov 02 23:06:13.044314 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:13.044265 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" podStartSLOduration=2.044251913 podStartE2EDuration="2.044251913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:06:13.039986883 +0000 UTC m=+1059.278484575" watchObservedRunningTime="2026-11-02 23:06:13.044251913 +0000 UTC m=+1059.282749576" Nov 02 23:06:13.475339 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:13.475299 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:13.477183 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:13.477148 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-gm8cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 23:06:13.477310 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:13.477250 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" podUID="4cb4d651-3373-4449-8f2f-0cc8f748b1b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 23:06:14.473567 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:14.473526 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-gm8cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 23:06:14.473887 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:14.473857 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" podUID="4cb4d651-3373-4449-8f2f-0cc8f748b1b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 23:06:15.473996 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:15.473531 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-gm8cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 23:06:15.473996 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:15.473608 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" podUID="4cb4d651-3373-4449-8f2f-0cc8f748b1b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 23:06:16.475705 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:16.475666 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-gm8cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 23:06:16.476259 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:16.475885 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" podUID="4cb4d651-3373-4449-8f2f-0cc8f748b1b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 23:06:17.475581 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:17.475542 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-gm8cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 23:06:17.475896 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:17.475868 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" podUID="4cb4d651-3373-4449-8f2f-0cc8f748b1b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 23:06:18.473653 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:18.473614 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-gm8cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 23:06:18.473845 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:18.473673 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" podUID="4cb4d651-3373-4449-8f2f-0cc8f748b1b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 23:06:19.473814 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:19.473776 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-gm8cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 23:06:19.474234 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:19.473852 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" podUID="4cb4d651-3373-4449-8f2f-0cc8f748b1b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 23:06:20.477837 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:20.477809 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:20.480109 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:20.478094 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:20.480109 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:06:20.478756 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:06:27.846215 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:06:27.846174 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:06:57.850001 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:06:57.849949 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:07:24.115576 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.115536 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6"] Nov 02 23:07:24.119909 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.119882 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:24.145206 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.145167 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:24.145405 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.145215 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57d13cfb-07f6-4709-8ceb-068e0528e721-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:24.145405 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.145322 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds99n\" (UniqueName: \"kubernetes.io/projected/57d13cfb-07f6-4709-8ceb-068e0528e721-kube-api-access-ds99n\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:24.145500 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.145476 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:24.145550 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.145511 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:24.145550 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.145533 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:24.145646 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.145550 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57d13cfb-07f6-4709-8ceb-068e0528e721-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:24.145646 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.145585 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57d13cfb-07f6-4709-8ceb-068e0528e721-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:24.145716 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.145654 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:24.228502 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.228461 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6"] Nov 02 23:07:24.246519 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.246477 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:24.246726 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.246544 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57d13cfb-07f6-4709-8ceb-068e0528e721-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:24.246726 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.246587 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds99n\" (UniqueName: \"kubernetes.io/projected/57d13cfb-07f6-4709-8ceb-068e0528e721-kube-api-access-ds99n\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:24.246846 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.246732 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:24.246846 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.246763 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:24.246846 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.246797 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:24.246846 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.246822 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57d13cfb-07f6-4709-8ceb-068e0528e721-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:24.246846 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.246846 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57d13cfb-07f6-4709-8ceb-068e0528e721-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:24.247108 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.246927 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:24.247108 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.246971 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:24.247559 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.247513 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:24.247559 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.247540 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:24.247752 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.247729 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57d13cfb-07f6-4709-8ceb-068e0528e721-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:24.248244 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.248206 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:24.249892 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.249859 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57d13cfb-07f6-4709-8ceb-068e0528e721-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:24.250001 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.249936 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:24.307585 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.307558 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57d13cfb-07f6-4709-8ceb-068e0528e721-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:24.308073 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.308049 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds99n\" (UniqueName: \"kubernetes.io/projected/57d13cfb-07f6-4709-8ceb-068e0528e721-kube-api-access-ds99n\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:24.438276 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.438187 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:24.878170 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.878146 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6"] Nov 02 23:07:24.879995 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:07:24.879969 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57d13cfb_07f6_4709_8ceb_068e0528e721.slice/crio-68cb10bf0afd4ae1de3cc0430fc1d5284a6f6ab7a9159db6f37013c19bc356a3 WatchSource:0}: Error finding container 68cb10bf0afd4ae1de3cc0430fc1d5284a6f6ab7a9159db6f37013c19bc356a3: Status 404 returned error can't find the container with id 68cb10bf0afd4ae1de3cc0430fc1d5284a6f6ab7a9159db6f37013c19bc356a3 Nov 02 23:07:24.882827 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.882790 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:07:24.882931 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.882874 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:07:24.883013 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:24.882926 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:07:25.435196 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:25.435159 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" event={"ID":"57d13cfb-07f6-4709-8ceb-068e0528e721","Type":"ContainerStarted","Data":"c0adaf38da9467dc2dac487c70aebe1217051f5e4728f67d32efe2602dce3eb4"} Nov 02 23:07:25.435642 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:25.435239 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" event={"ID":"57d13cfb-07f6-4709-8ceb-068e0528e721","Type":"ContainerStarted","Data":"68cb10bf0afd4ae1de3cc0430fc1d5284a6f6ab7a9159db6f37013c19bc356a3"} Nov 02 23:07:25.438637 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:25.438619 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:25.440593 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:25.440569 5727 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 23:07:25.440714 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:25.440617 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" podUID="57d13cfb-07f6-4709-8ceb-068e0528e721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 23:07:25.589279 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:25.589210 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" podStartSLOduration=1.589188494 podStartE2EDuration="1.589188494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:07:25.555622723 +0000 UTC m=+1131.794120410" watchObservedRunningTime="2026-11-02 23:07:25.589188494 +0000 UTC m=+1131.827686157" Nov 02 23:07:26.438876 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:26.438838 5727 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 23:07:26.439304 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:26.438905 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" podUID="57d13cfb-07f6-4709-8ceb-068e0528e721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 23:07:27.438724 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:27.438690 5727 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 23:07:27.438940 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:27.438741 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" podUID="57d13cfb-07f6-4709-8ceb-068e0528e721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 23:07:27.846734 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:07:27.846696 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:07:28.438549 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:28.438513 5727 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 23:07:28.438804 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:28.438574 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" podUID="57d13cfb-07f6-4709-8ceb-068e0528e721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 23:07:29.439431 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:29.439394 5727 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 23:07:29.439843 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:29.439448 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" podUID="57d13cfb-07f6-4709-8ceb-068e0528e721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 23:07:30.438917 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:30.438880 5727 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 23:07:30.439092 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:30.438951 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" podUID="57d13cfb-07f6-4709-8ceb-068e0528e721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 23:07:31.438869 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:31.438828 5727 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 23:07:31.439400 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:31.438909 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" podUID="57d13cfb-07f6-4709-8ceb-068e0528e721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 23:07:32.444185 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:32.444156 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:32.444669 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:32.444649 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:32.445404 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:07:32.445383 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:07:57.846538 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:07:57.846500 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:08:11.603595 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:11.603563 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q"] Nov 02 23:08:11.607833 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:11.607816 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" Nov 02 23:08:11.710888 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:11.710858 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b42983\"/\"distinct-2-w1-b42983-kserve-self-signed-certs\"" Nov 02 23:08:11.716248 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:11.716214 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q"] Nov 02 23:08:11.722873 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:11.722838 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3799f21-d78b-468d-9347-ca3f0968409d-model-cache\") pod \"distinct-2-w1-b42983-kserve-b65669645-72g6q\" (UID: \"d3799f21-d78b-468d-9347-ca3f0968409d\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" Nov 02 23:08:11.723186 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:11.723161 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md42w\" (UniqueName: \"kubernetes.io/projected/d3799f21-d78b-468d-9347-ca3f0968409d-kube-api-access-md42w\") pod \"distinct-2-w1-b42983-kserve-b65669645-72g6q\" (UID: \"d3799f21-d78b-468d-9347-ca3f0968409d\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" Nov 02 23:08:11.723326 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:11.723311 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3799f21-d78b-468d-9347-ca3f0968409d-home\") pod \"distinct-2-w1-b42983-kserve-b65669645-72g6q\" (UID: \"d3799f21-d78b-468d-9347-ca3f0968409d\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" Nov 02 23:08:11.723451 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:11.723437 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3799f21-d78b-468d-9347-ca3f0968409d-tmp-dir\") pod \"distinct-2-w1-b42983-kserve-b65669645-72g6q\" (UID: \"d3799f21-d78b-468d-9347-ca3f0968409d\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" Nov 02 23:08:11.723599 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:11.723582 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3799f21-d78b-468d-9347-ca3f0968409d-dshm\") pod \"distinct-2-w1-b42983-kserve-b65669645-72g6q\" (UID: \"d3799f21-d78b-468d-9347-ca3f0968409d\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" Nov 02 23:08:11.723804 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:11.723786 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3799f21-d78b-468d-9347-ca3f0968409d-tls-certs\") pod \"distinct-2-w1-b42983-kserve-b65669645-72g6q\" (UID: \"d3799f21-d78b-468d-9347-ca3f0968409d\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" Nov 02 23:08:11.824763 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:11.824732 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3799f21-d78b-468d-9347-ca3f0968409d-tls-certs\") pod \"distinct-2-w1-b42983-kserve-b65669645-72g6q\" (UID: \"d3799f21-d78b-468d-9347-ca3f0968409d\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" Nov 02 23:08:11.824941 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:11.824793 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3799f21-d78b-468d-9347-ca3f0968409d-model-cache\") pod \"distinct-2-w1-b42983-kserve-b65669645-72g6q\" (UID: \"d3799f21-d78b-468d-9347-ca3f0968409d\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" Nov 02 23:08:11.824941 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:11.824863 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md42w\" (UniqueName: \"kubernetes.io/projected/d3799f21-d78b-468d-9347-ca3f0968409d-kube-api-access-md42w\") pod \"distinct-2-w1-b42983-kserve-b65669645-72g6q\" (UID: \"d3799f21-d78b-468d-9347-ca3f0968409d\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" Nov 02 23:08:11.824941 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:11.824892 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3799f21-d78b-468d-9347-ca3f0968409d-home\") pod \"distinct-2-w1-b42983-kserve-b65669645-72g6q\" (UID: \"d3799f21-d78b-468d-9347-ca3f0968409d\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" Nov 02 23:08:11.824941 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:11.824929 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3799f21-d78b-468d-9347-ca3f0968409d-tmp-dir\") pod \"distinct-2-w1-b42983-kserve-b65669645-72g6q\" (UID: \"d3799f21-d78b-468d-9347-ca3f0968409d\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" Nov 02 23:08:11.825103 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:11.824964 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3799f21-d78b-468d-9347-ca3f0968409d-dshm\") pod \"distinct-2-w1-b42983-kserve-b65669645-72g6q\" (UID: \"d3799f21-d78b-468d-9347-ca3f0968409d\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" Nov 02 23:08:11.825304 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:11.825281 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3799f21-d78b-468d-9347-ca3f0968409d-model-cache\") pod \"distinct-2-w1-b42983-kserve-b65669645-72g6q\" (UID: \"d3799f21-d78b-468d-9347-ca3f0968409d\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" Nov 02 23:08:11.825432 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:11.825334 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3799f21-d78b-468d-9347-ca3f0968409d-home\") pod \"distinct-2-w1-b42983-kserve-b65669645-72g6q\" (UID: \"d3799f21-d78b-468d-9347-ca3f0968409d\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" Nov 02 23:08:11.825432 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:11.825379 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3799f21-d78b-468d-9347-ca3f0968409d-tmp-dir\") pod \"distinct-2-w1-b42983-kserve-b65669645-72g6q\" (UID: \"d3799f21-d78b-468d-9347-ca3f0968409d\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" Nov 02 23:08:11.827524 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:11.827491 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3799f21-d78b-468d-9347-ca3f0968409d-dshm\") pod \"distinct-2-w1-b42983-kserve-b65669645-72g6q\" (UID: \"d3799f21-d78b-468d-9347-ca3f0968409d\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" Nov 02 23:08:11.827647 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:11.827628 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3799f21-d78b-468d-9347-ca3f0968409d-tls-certs\") pod \"distinct-2-w1-b42983-kserve-b65669645-72g6q\" (UID: \"d3799f21-d78b-468d-9347-ca3f0968409d\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" Nov 02 23:08:11.889875 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:11.889797 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md42w\" (UniqueName: \"kubernetes.io/projected/d3799f21-d78b-468d-9347-ca3f0968409d-kube-api-access-md42w\") pod \"distinct-2-w1-b42983-kserve-b65669645-72g6q\" (UID: \"d3799f21-d78b-468d-9347-ca3f0968409d\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" Nov 02 23:08:11.919819 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:11.919791 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" Nov 02 23:08:12.360748 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:12.360722 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q"] Nov 02 23:08:12.813734 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:12.813697 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" event={"ID":"d3799f21-d78b-468d-9347-ca3f0968409d","Type":"ContainerStarted","Data":"0b7a6bb7b7d8fa768146f7a64038da011c55643970d36d0a2789a74b691349e1"} Nov 02 23:08:12.814108 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:12.813748 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" event={"ID":"d3799f21-d78b-468d-9347-ca3f0968409d","Type":"ContainerStarted","Data":"f8040b5e4a0eb96cd9505aa72a919f0f8dad9c7904784b924954e55a23d31b2e"} Nov 02 23:08:12.920723 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:12.920693 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" Nov 02 23:08:12.933535 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:12.933477 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" podStartSLOduration=1.933458458 podStartE2EDuration="1.933458458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:08:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:08:12.928857105 +0000 UTC m=+1179.167354781" watchObservedRunningTime="2026-11-02 23:08:12.933458458 +0000 UTC m=+1179.171956121" Nov 02 23:08:21.920004 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:21.919967 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" Nov 02 23:08:21.932822 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:21.932801 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" Nov 02 23:08:22.898019 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:22.897991 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" Nov 02 23:08:26.604253 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:26.604205 5727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" podUID="f83c8651-5db1-4e80-b9fb-ec5b793e7e94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:08:26.893328 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:26.892215 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" podUID="d3799f21-d78b-468d-9347-ca3f0968409d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:08:26.914097 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:26.913219 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" podUID="4b724650-f2c7-4a4b-a077-edf537b42392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:08:27.215314 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:27.215212 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" podUID="f83c8651-5db1-4e80-b9fb-ec5b793e7e94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:08:27.905154 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:08:27.904308 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:08:34.277281 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:34.277254 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 02 23:08:34.303635 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:34.303607 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 23:08:34.303854 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:34.303835 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 02 23:08:34.318024 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:08:34.318005 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 23:08:57.846309 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:08:57.846269 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:09:04.675995 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:04.675966 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs"] Nov 02 23:09:04.680243 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:04.680219 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:04.746867 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:04.746832 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfdaf98c-fc85-4457-8366-4a81aab5573a-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:04.746867 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:04.746871 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfdaf98c-fc85-4457-8366-4a81aab5573a-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:04.747121 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:04.746888 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:04.747121 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:04.747028 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:04.747121 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:04.747063 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:04.747121 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:04.747106 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfdaf98c-fc85-4457-8366-4a81aab5573a-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:04.747335 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:04.747183 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:04.747335 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:04.747286 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:04.747335 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:04.747331 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plmh5\" (UniqueName: \"kubernetes.io/projected/bfdaf98c-fc85-4457-8366-4a81aab5573a-kube-api-access-plmh5\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:04.787410 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:04.787372 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs"] Nov 02 23:09:04.848915 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:04.848881 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfdaf98c-fc85-4457-8366-4a81aab5573a-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:04.848915 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:04.848917 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:04.849186 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:04.848951 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:04.849186 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:04.848969 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:04.849186 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:04.849005 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfdaf98c-fc85-4457-8366-4a81aab5573a-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:04.849186 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:04.849022 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:04.849186 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:04.849095 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:04.849186 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:04.849173 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plmh5\" (UniqueName: \"kubernetes.io/projected/bfdaf98c-fc85-4457-8366-4a81aab5573a-kube-api-access-plmh5\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:04.849515 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:04.849293 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfdaf98c-fc85-4457-8366-4a81aab5573a-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:04.849515 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:04.849411 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:04.849515 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:04.849486 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:04.849515 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:04.849509 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:04.849719 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:04.849699 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:04.850019 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:04.850001 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfdaf98c-fc85-4457-8366-4a81aab5573a-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:04.852047 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:04.852026 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:04.852279 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:04.852261 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfdaf98c-fc85-4457-8366-4a81aab5573a-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:04.910638 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:04.910608 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfdaf98c-fc85-4457-8366-4a81aab5573a-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:04.910805 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:04.910785 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plmh5\" (UniqueName: \"kubernetes.io/projected/bfdaf98c-fc85-4457-8366-4a81aab5573a-kube-api-access-plmh5\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:04.996759 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:04.996678 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:05.427108 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:05.427084 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs"] Nov 02 23:09:05.428190 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:09:05.428166 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfdaf98c_fc85_4457_8366_4a81aab5573a.slice/crio-4d1b942326ddb82a0cc157deeda02a4a946b8d3fc9067c3ab54b10d0be3738af WatchSource:0}: Error finding container 4d1b942326ddb82a0cc157deeda02a4a946b8d3fc9067c3ab54b10d0be3738af: Status 404 returned error can't find the container with id 4d1b942326ddb82a0cc157deeda02a4a946b8d3fc9067c3ab54b10d0be3738af Nov 02 23:09:05.430613 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:05.430581 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:09:05.430719 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:05.430669 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:09:05.430783 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:05.430715 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:09:06.384715 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:06.384676 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" event={"ID":"bfdaf98c-fc85-4457-8366-4a81aab5573a","Type":"ContainerStarted","Data":"d5286ad12669a017e9a8891815dcbe109d13f8e1a6f3659f4649b36298c0fe27"} Nov 02 23:09:06.385153 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:06.384736 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" event={"ID":"bfdaf98c-fc85-4457-8366-4a81aab5573a","Type":"ContainerStarted","Data":"4d1b942326ddb82a0cc157deeda02a4a946b8d3fc9067c3ab54b10d0be3738af"} Nov 02 23:09:06.576860 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:06.576793 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" podStartSLOduration=2.5767725219999997 podStartE2EDuration="2.576772522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:09:06.559584409 +0000 UTC m=+1232.798082094" watchObservedRunningTime="2026-11-02 23:09:06.576772522 +0000 UTC m=+1232.815270185" Nov 02 23:09:06.997720 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:06.997689 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:06.999751 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:06.999722 5727 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 23:09:06.999887 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:06.999776 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" podUID="bfdaf98c-fc85-4457-8366-4a81aab5573a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 23:09:07.997537 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:07.997492 5727 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 23:09:07.997928 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:07.997552 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" podUID="bfdaf98c-fc85-4457-8366-4a81aab5573a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 23:09:08.998007 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:08.997970 5727 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 23:09:08.998548 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:08.998029 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" podUID="bfdaf98c-fc85-4457-8366-4a81aab5573a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 23:09:10.000181 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:09.999714 5727 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 23:09:10.000181 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:09.999780 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" podUID="bfdaf98c-fc85-4457-8366-4a81aab5573a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 23:09:10.997210 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:10.997177 5727 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 23:09:10.997388 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:10.997233 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" podUID="bfdaf98c-fc85-4457-8366-4a81aab5573a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 23:09:11.997677 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:11.997642 5727 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 23:09:11.998052 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:11.997697 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" podUID="bfdaf98c-fc85-4457-8366-4a81aab5573a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 23:09:13.017516 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:13.017480 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:13.018695 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:13.018659 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:13.020252 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:13.020092 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:09:18.290750 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:18.290697 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r"] Nov 02 23:09:18.295708 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:18.295682 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:18.399940 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:18.399900 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae279e6-131e-446b-9ad5-6c69582fa70f-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:18.400215 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:18.400007 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:18.400215 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:18.400053 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:18.400215 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:18.400079 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:18.400215 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:18.400159 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:18.400215 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:18.400190 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:18.400647 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:18.400232 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae279e6-131e-446b-9ad5-6c69582fa70f-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:18.400647 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:18.400324 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae279e6-131e-446b-9ad5-6c69582fa70f-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:18.400647 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:18.400384 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjqtt\" (UniqueName: \"kubernetes.io/projected/2ae279e6-131e-446b-9ad5-6c69582fa70f-kube-api-access-hjqtt\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:18.403741 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:18.403712 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r"] Nov 02 23:09:18.501919 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:18.501887 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae279e6-131e-446b-9ad5-6c69582fa70f-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:18.502080 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:18.501935 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:18.502080 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:18.502063 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:18.502182 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:18.502095 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:18.502237 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:18.502188 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:18.502237 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:18.502218 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:18.502340 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:18.502263 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae279e6-131e-446b-9ad5-6c69582fa70f-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:18.502397 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:18.502353 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:18.502397 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:18.502359 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae279e6-131e-446b-9ad5-6c69582fa70f-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:18.502513 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:18.502450 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjqtt\" (UniqueName: \"kubernetes.io/projected/2ae279e6-131e-446b-9ad5-6c69582fa70f-kube-api-access-hjqtt\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:18.502513 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:18.502469 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:18.502621 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:18.502549 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:18.502621 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:18.502583 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:18.503184 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:18.503154 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae279e6-131e-446b-9ad5-6c69582fa70f-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:18.504797 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:18.504769 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:18.505140 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:18.505108 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae279e6-131e-446b-9ad5-6c69582fa70f-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:18.565019 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:18.564986 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae279e6-131e-446b-9ad5-6c69582fa70f-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:18.565183 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:18.565109 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjqtt\" (UniqueName: \"kubernetes.io/projected/2ae279e6-131e-446b-9ad5-6c69582fa70f-kube-api-access-hjqtt\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:18.610013 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:18.609984 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:19.043770 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:19.043740 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r"] Nov 02 23:09:19.044923 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:09:19.044903 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae279e6_131e_446b_9ad5_6c69582fa70f.slice/crio-8b856742f0ce9357283a5094ad62a86cd5ea23156e0dfcd17ff3a52e7b9746c2 WatchSource:0}: Error finding container 8b856742f0ce9357283a5094ad62a86cd5ea23156e0dfcd17ff3a52e7b9746c2: Status 404 returned error can't find the container with id 8b856742f0ce9357283a5094ad62a86cd5ea23156e0dfcd17ff3a52e7b9746c2 Nov 02 23:09:19.047244 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:19.047213 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:09:19.047338 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:19.047273 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:09:19.047338 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:19.047303 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:09:19.593167 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:19.592144 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" event={"ID":"2ae279e6-131e-446b-9ad5-6c69582fa70f","Type":"ContainerStarted","Data":"311762f5998ee29aa53844c2e13dbe3bc0c8cf788bd7ca0074629d72e4507cf4"} Nov 02 23:09:19.593167 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:19.592204 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" event={"ID":"2ae279e6-131e-446b-9ad5-6c69582fa70f","Type":"ContainerStarted","Data":"8b856742f0ce9357283a5094ad62a86cd5ea23156e0dfcd17ff3a52e7b9746c2"} Nov 02 23:09:19.610581 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:19.610554 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:19.712873 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:19.712820 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" podStartSLOduration=1.712806345 podStartE2EDuration="1.712806345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:09:19.71022994 +0000 UTC m=+1245.948727653" watchObservedRunningTime="2026-11-02 23:09:19.712806345 +0000 UTC m=+1245.951304006" Nov 02 23:09:20.613048 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:20.613018 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 02 23:09:20.613482 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:20.613065 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" podUID="2ae279e6-131e-446b-9ad5-6c69582fa70f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 02 23:09:21.187785 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:21.187747 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7"] Nov 02 23:09:21.610810 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:21.610773 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 02 23:09:21.610993 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:21.610827 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" podUID="2ae279e6-131e-446b-9ad5-6c69582fa70f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 02 23:09:22.610917 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:22.610878 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 02 23:09:22.611325 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:22.610941 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" podUID="2ae279e6-131e-446b-9ad5-6c69582fa70f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 02 23:09:23.610941 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:23.610901 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 02 23:09:23.611356 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:23.610958 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" podUID="2ae279e6-131e-446b-9ad5-6c69582fa70f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 02 23:09:24.610825 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:24.610791 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 02 23:09:24.611003 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:24.610845 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" podUID="2ae279e6-131e-446b-9ad5-6c69582fa70f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 02 23:09:25.610833 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:25.610799 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 02 23:09:25.611003 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:25.610852 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" podUID="2ae279e6-131e-446b-9ad5-6c69582fa70f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 02 23:09:26.620530 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:26.620484 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:26.625484 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:26.625459 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:26.642480 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:26.642434 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:09:27.907557 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:09:27.907471 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:09:36.209101 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:36.209062 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" podUID="4b724650-f2c7-4a4b-a077-edf537b42392" containerName="main" containerID="cri-o://4c22d10d5eacf39b3557d51331917bc81d2f4521f0748b1cc3bf9ddb5dfdba72" gracePeriod=45 Nov 02 23:09:36.507111 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:36.507086 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" Nov 02 23:09:36.647748 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:36.647715 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b724650-f2c7-4a4b-a077-edf537b42392-kube-api-access-mr9g6\" (UniqueName: \"kubernetes.io/projected/4b724650-f2c7-4a4b-a077-edf537b42392-kube-api-access-mr9g6\") pod \"4b724650-f2c7-4a4b-a077-edf537b42392\" (UID: \"4b724650-f2c7-4a4b-a077-edf537b42392\") " Nov 02 23:09:36.647947 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:36.647785 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b724650-f2c7-4a4b-a077-edf537b42392-tls-certs\" (UniqueName: \"kubernetes.io/secret/4b724650-f2c7-4a4b-a077-edf537b42392-tls-certs\") pod \"4b724650-f2c7-4a4b-a077-edf537b42392\" (UID: \"4b724650-f2c7-4a4b-a077-edf537b42392\") " Nov 02 23:09:36.647947 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:36.647847 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b724650-f2c7-4a4b-a077-edf537b42392-home\" (UniqueName: \"kubernetes.io/empty-dir/4b724650-f2c7-4a4b-a077-edf537b42392-home\") pod \"4b724650-f2c7-4a4b-a077-edf537b42392\" (UID: \"4b724650-f2c7-4a4b-a077-edf537b42392\") " Nov 02 23:09:36.648077 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:36.647992 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b724650-f2c7-4a4b-a077-edf537b42392-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b724650-f2c7-4a4b-a077-edf537b42392-model-cache\") pod \"4b724650-f2c7-4a4b-a077-edf537b42392\" (UID: \"4b724650-f2c7-4a4b-a077-edf537b42392\") " Nov 02 23:09:36.648077 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:36.648025 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b724650-f2c7-4a4b-a077-edf537b42392-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b724650-f2c7-4a4b-a077-edf537b42392-dshm\") pod \"4b724650-f2c7-4a4b-a077-edf537b42392\" (UID: \"4b724650-f2c7-4a4b-a077-edf537b42392\") " Nov 02 23:09:36.648077 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:36.648051 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b724650-f2c7-4a4b-a077-edf537b42392-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b724650-f2c7-4a4b-a077-edf537b42392-tmp-dir\") pod \"4b724650-f2c7-4a4b-a077-edf537b42392\" (UID: \"4b724650-f2c7-4a4b-a077-edf537b42392\") " Nov 02 23:09:36.649020 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:36.648362 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b724650-f2c7-4a4b-a077-edf537b42392-model-cache" pod "4b724650-f2c7-4a4b-a077-edf537b42392" (UID: "4b724650-f2c7-4a4b-a077-edf537b42392"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:09:36.649020 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:36.648565 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b724650-f2c7-4a4b-a077-edf537b42392-tmp-dir" pod "4b724650-f2c7-4a4b-a077-edf537b42392" (UID: "4b724650-f2c7-4a4b-a077-edf537b42392"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:09:36.649020 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:36.648739 5727 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b724650-f2c7-4a4b-a077-edf537b42392-model-cache\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:36.649020 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:36.648762 5727 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b724650-f2c7-4a4b-a077-edf537b42392-tmp-dir\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:36.649020 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:36.648993 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b724650-f2c7-4a4b-a077-edf537b42392-home" pod "4b724650-f2c7-4a4b-a077-edf537b42392" (UID: "4b724650-f2c7-4a4b-a077-edf537b42392"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:09:36.650593 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:36.650570 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b724650-f2c7-4a4b-a077-edf537b42392-dshm" pod "4b724650-f2c7-4a4b-a077-edf537b42392" (UID: "4b724650-f2c7-4a4b-a077-edf537b42392"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:09:36.650793 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:36.650762 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b724650-f2c7-4a4b-a077-edf537b42392-tls-certs" pod "4b724650-f2c7-4a4b-a077-edf537b42392" (UID: "4b724650-f2c7-4a4b-a077-edf537b42392"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:09:36.650869 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:36.650832 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b724650-f2c7-4a4b-a077-edf537b42392-kube-api-access-mr9g6" pod "4b724650-f2c7-4a4b-a077-edf537b42392" (UID: "4b724650-f2c7-4a4b-a077-edf537b42392"). InnerVolumeSpecName "kube-api-access-mr9g6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:09:36.749699 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:36.749624 5727 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b724650-f2c7-4a4b-a077-edf537b42392-home\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:36.749699 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:36.749649 5727 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b724650-f2c7-4a4b-a077-edf537b42392-dshm\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:36.749699 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:36.749659 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr9g6\" (UniqueName: \"kubernetes.io/projected/4b724650-f2c7-4a4b-a077-edf537b42392-kube-api-access-mr9g6\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:36.749699 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:36.749669 5727 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b724650-f2c7-4a4b-a077-edf537b42392-tls-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:36.775179 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:36.775147 5727 generic.go:356] "Generic (PLEG): container finished" podID="4b724650-f2c7-4a4b-a077-edf537b42392" containerID="4c22d10d5eacf39b3557d51331917bc81d2f4521f0748b1cc3bf9ddb5dfdba72" exitCode=0 Nov 02 23:09:36.775364 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:36.775187 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" Nov 02 23:09:36.775364 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:36.775220 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" event={"ID":"4b724650-f2c7-4a4b-a077-edf537b42392","Type":"ContainerDied","Data":"4c22d10d5eacf39b3557d51331917bc81d2f4521f0748b1cc3bf9ddb5dfdba72"} Nov 02 23:09:36.775364 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:36.775265 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7" event={"ID":"4b724650-f2c7-4a4b-a077-edf537b42392","Type":"ContainerDied","Data":"2785dc1e11fda60336489f3639900227b2e84da3e3e4983349034c5027b327e8"} Nov 02 23:09:36.775364 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:36.775288 5727 scope.go:122] "RemoveContainer" containerID="4c22d10d5eacf39b3557d51331917bc81d2f4521f0748b1cc3bf9ddb5dfdba72" Nov 02 23:09:36.788047 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:36.788028 5727 scope.go:122] "RemoveContainer" containerID="4c22d10d5eacf39b3557d51331917bc81d2f4521f0748b1cc3bf9ddb5dfdba72" Nov 02 23:09:36.788364 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:09:36.788344 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c22d10d5eacf39b3557d51331917bc81d2f4521f0748b1cc3bf9ddb5dfdba72\": container with ID starting with 4c22d10d5eacf39b3557d51331917bc81d2f4521f0748b1cc3bf9ddb5dfdba72 not found: ID does not exist" containerID="4c22d10d5eacf39b3557d51331917bc81d2f4521f0748b1cc3bf9ddb5dfdba72" Nov 02 23:09:36.788478 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:36.788372 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c22d10d5eacf39b3557d51331917bc81d2f4521f0748b1cc3bf9ddb5dfdba72"} err="failed to get container status \"4c22d10d5eacf39b3557d51331917bc81d2f4521f0748b1cc3bf9ddb5dfdba72\": rpc error: code = NotFound desc = could not find container \"4c22d10d5eacf39b3557d51331917bc81d2f4521f0748b1cc3bf9ddb5dfdba72\": container with ID starting with 4c22d10d5eacf39b3557d51331917bc81d2f4521f0748b1cc3bf9ddb5dfdba72 not found: ID does not exist" Nov 02 23:09:36.939463 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:36.939431 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7"] Nov 02 23:09:36.948589 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:36.948567 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-a-4cba9672-kserve-696d868cb9-nlqz7"] Nov 02 23:09:37.883611 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:37.883580 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b724650-f2c7-4a4b-a077-edf537b42392" path="/var/lib/kubelet/pods/4b724650-f2c7-4a4b-a077-edf537b42392/volumes" Nov 02 23:09:40.101153 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:40.100238 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:09:40.101153 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:40.100316 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" podUID="575d202b-f514-4312-8dc7-dfad51c6ca19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:09:51.140801 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.140698 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7"] Nov 02 23:09:51.141727 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.141701 5727 cpu_manager.go:414] "containerMap: removing container" podUID="4b724650-f2c7-4a4b-a077-edf537b42392" containerName="main" Nov 02 23:09:51.141833 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.141728 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b724650-f2c7-4a4b-a077-edf537b42392" containerName="main" Nov 02 23:09:51.141960 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.141943 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b724650-f2c7-4a4b-a077-edf537b42392" containerName="main" Nov 02 23:09:51.146119 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.146096 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:09:51.201239 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.201192 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-96031a-openshift-default-dockercfg-fx8gh\"" Nov 02 23:09:51.202050 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.202023 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istio-data\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:09:51.202250 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.202230 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-workload-certs\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:09:51.202384 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.202368 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-workload-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:09:51.202514 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.202497 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7psl\" (UniqueName: \"kubernetes.io/projected/61ae60a1-87a0-42bf-9342-0420cd1cbce7-kube-api-access-k7psl\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:09:51.202639 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.202625 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istio-token\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:09:51.202760 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.202744 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istio-podinfo\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:09:51.202873 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.202857 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-credential-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:09:51.203041 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.203025 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istio-envoy\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:09:51.203232 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.203212 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istiod-ca-cert\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:09:51.274157 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.256534 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7"] Nov 02 23:09:51.317915 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.317286 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istio-envoy\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:09:51.318384 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.318362 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istiod-ca-cert\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:09:51.352973 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.329109 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istiod-ca-cert\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:09:51.354333 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.354308 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istio-data\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:09:51.354858 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.354759 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istio-data\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:09:51.355017 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.354998 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-workload-certs\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:09:51.355525 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.355507 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-workload-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:09:51.361421 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.361397 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7psl\" (UniqueName: \"kubernetes.io/projected/61ae60a1-87a0-42bf-9342-0420cd1cbce7-kube-api-access-k7psl\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:09:51.361910 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.355412 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-workload-certs\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:09:51.362053 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.355918 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-workload-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:09:51.362218 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.362201 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istio-token\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:09:51.481480 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.481385 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istio-podinfo\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:09:51.482193 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.482170 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-credential-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:09:51.482774 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.482584 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-credential-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:09:51.492810 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.492784 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istio-podinfo\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:09:51.498726 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.497772 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7psl\" (UniqueName: \"kubernetes.io/projected/61ae60a1-87a0-42bf-9342-0420cd1cbce7-kube-api-access-k7psl\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:09:51.537163 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.506404 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istio-envoy\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:09:51.636199 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.619047 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istio-token\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:09:51.773202 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:51.772185 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:09:52.808103 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:52.808073 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7"] Nov 02 23:09:52.809354 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:09:52.809326 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ae60a1_87a0_42bf_9342_0420cd1cbce7.slice/crio-0398ead25e66aa40304792c015bff44cef9875165f3c4814ba0229735530c435 WatchSource:0}: Error finding container 0398ead25e66aa40304792c015bff44cef9875165f3c4814ba0229735530c435: Status 404 returned error can't find the container with id 0398ead25e66aa40304792c015bff44cef9875165f3c4814ba0229735530c435 Nov 02 23:09:52.811732 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:52.811702 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:09:52.811801 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:52.811766 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:09:52.811801 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:52.811795 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:09:53.168016 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:53.167982 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" event={"ID":"61ae60a1-87a0-42bf-9342-0420cd1cbce7","Type":"ContainerStarted","Data":"0f2ce3aae9ac1f1c50751326a8dfdb4d14dbfffefecb0ab82ad132739c82c9a0"} Nov 02 23:09:53.168236 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:53.168037 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" event={"ID":"61ae60a1-87a0-42bf-9342-0420cd1cbce7","Type":"ContainerStarted","Data":"0398ead25e66aa40304792c015bff44cef9875165f3c4814ba0229735530c435"} Nov 02 23:09:53.301443 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:53.301388 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" podStartSLOduration=2.3013736590000002 podStartE2EDuration="2.301373659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:09:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:09:53.298740811 +0000 UTC m=+1279.537238472" watchObservedRunningTime="2026-11-02 23:09:53.301373659 +0000 UTC m=+1279.539871320" Nov 02 23:09:53.775306 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:53.775267 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:09:53.777404 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:53.777373 5727 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 23:09:53.777555 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:53.777439 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" podUID="61ae60a1-87a0-42bf-9342-0420cd1cbce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 23:09:54.775598 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:54.775551 5727 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 23:09:54.776104 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:54.775871 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" podUID="61ae60a1-87a0-42bf-9342-0420cd1cbce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 23:09:55.775617 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:55.775576 5727 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 23:09:55.776246 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:55.776213 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" podUID="61ae60a1-87a0-42bf-9342-0420cd1cbce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 23:09:56.234718 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:56.234685 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn"] Nov 02 23:09:56.238884 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:56.238862 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" Nov 02 23:09:56.296194 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:56.296151 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ef6e83d-2b3b-44c2-a973-86f2587ac616-dshm\") pod \"distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn\" (UID: \"7ef6e83d-2b3b-44c2-a973-86f2587ac616\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" Nov 02 23:09:56.296506 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:56.296485 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkvwp\" (UniqueName: \"kubernetes.io/projected/7ef6e83d-2b3b-44c2-a973-86f2587ac616-kube-api-access-kkvwp\") pod \"distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn\" (UID: \"7ef6e83d-2b3b-44c2-a973-86f2587ac616\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" Nov 02 23:09:56.296786 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:56.296766 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ef6e83d-2b3b-44c2-a973-86f2587ac616-home\") pod \"distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn\" (UID: \"7ef6e83d-2b3b-44c2-a973-86f2587ac616\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" Nov 02 23:09:56.296915 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:56.296897 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ef6e83d-2b3b-44c2-a973-86f2587ac616-model-cache\") pod \"distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn\" (UID: \"7ef6e83d-2b3b-44c2-a973-86f2587ac616\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" Nov 02 23:09:56.297019 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:56.297006 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ef6e83d-2b3b-44c2-a973-86f2587ac616-tmp-dir\") pod \"distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn\" (UID: \"7ef6e83d-2b3b-44c2-a973-86f2587ac616\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" Nov 02 23:09:56.297197 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:56.297181 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ef6e83d-2b3b-44c2-a973-86f2587ac616-tls-certs\") pod \"distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn\" (UID: \"7ef6e83d-2b3b-44c2-a973-86f2587ac616\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" Nov 02 23:09:56.342387 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:56.342354 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-08c2c6\"/\"distinct-w2-08c2c6-kserve-self-signed-certs\"" Nov 02 23:09:56.347009 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:56.342741 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-08c2c6\"/\"openshift-service-ca.crt\"" Nov 02 23:09:56.347269 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:56.342865 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-08c2c6\"/\"default-dockercfg-zg2jb\"" Nov 02 23:09:56.347381 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:56.346938 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-08c2c6\"/\"kube-root-ca.crt\"" Nov 02 23:09:56.370639 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:56.370598 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn"] Nov 02 23:09:56.398207 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:56.398175 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ef6e83d-2b3b-44c2-a973-86f2587ac616-dshm\") pod \"distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn\" (UID: \"7ef6e83d-2b3b-44c2-a973-86f2587ac616\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" Nov 02 23:09:56.398471 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:56.398452 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkvwp\" (UniqueName: \"kubernetes.io/projected/7ef6e83d-2b3b-44c2-a973-86f2587ac616-kube-api-access-kkvwp\") pod \"distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn\" (UID: \"7ef6e83d-2b3b-44c2-a973-86f2587ac616\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" Nov 02 23:09:56.398661 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:56.398649 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ef6e83d-2b3b-44c2-a973-86f2587ac616-home\") pod \"distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn\" (UID: \"7ef6e83d-2b3b-44c2-a973-86f2587ac616\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" Nov 02 23:09:56.398758 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:56.398746 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ef6e83d-2b3b-44c2-a973-86f2587ac616-model-cache\") pod \"distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn\" (UID: \"7ef6e83d-2b3b-44c2-a973-86f2587ac616\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" Nov 02 23:09:56.398835 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:56.398825 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ef6e83d-2b3b-44c2-a973-86f2587ac616-tmp-dir\") pod \"distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn\" (UID: \"7ef6e83d-2b3b-44c2-a973-86f2587ac616\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" Nov 02 23:09:56.398966 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:56.398938 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ef6e83d-2b3b-44c2-a973-86f2587ac616-tls-certs\") pod \"distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn\" (UID: \"7ef6e83d-2b3b-44c2-a973-86f2587ac616\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" Nov 02 23:09:56.400441 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:56.400190 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ef6e83d-2b3b-44c2-a973-86f2587ac616-model-cache\") pod \"distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn\" (UID: \"7ef6e83d-2b3b-44c2-a973-86f2587ac616\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" Nov 02 23:09:56.400441 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:56.400406 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ef6e83d-2b3b-44c2-a973-86f2587ac616-tmp-dir\") pod \"distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn\" (UID: \"7ef6e83d-2b3b-44c2-a973-86f2587ac616\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" Nov 02 23:09:56.400564 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:56.400467 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ef6e83d-2b3b-44c2-a973-86f2587ac616-home\") pod \"distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn\" (UID: \"7ef6e83d-2b3b-44c2-a973-86f2587ac616\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" Nov 02 23:09:56.403288 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:56.403264 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ef6e83d-2b3b-44c2-a973-86f2587ac616-dshm\") pod \"distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn\" (UID: \"7ef6e83d-2b3b-44c2-a973-86f2587ac616\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" Nov 02 23:09:56.404246 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:56.404222 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ef6e83d-2b3b-44c2-a973-86f2587ac616-tls-certs\") pod \"distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn\" (UID: \"7ef6e83d-2b3b-44c2-a973-86f2587ac616\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" Nov 02 23:09:56.471229 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:56.471194 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkvwp\" (UniqueName: \"kubernetes.io/projected/7ef6e83d-2b3b-44c2-a973-86f2587ac616-kube-api-access-kkvwp\") pod \"distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn\" (UID: \"7ef6e83d-2b3b-44c2-a973-86f2587ac616\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" Nov 02 23:09:56.577154 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:56.575656 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" Nov 02 23:09:56.783152 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:56.779724 5727 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 23:09:56.783152 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:56.779804 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" podUID="61ae60a1-87a0-42bf-9342-0420cd1cbce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 23:09:57.027562 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:57.025773 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn"] Nov 02 23:09:57.029734 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:09:57.029615 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ef6e83d_2b3b_44c2_a973_86f2587ac616.slice/crio-21fd918261c4a9807af472c1616f3e8baa2d95eca9b0fc0078a13f8bc3797877 WatchSource:0}: Error finding container 21fd918261c4a9807af472c1616f3e8baa2d95eca9b0fc0078a13f8bc3797877: Status 404 returned error can't find the container with id 21fd918261c4a9807af472c1616f3e8baa2d95eca9b0fc0078a13f8bc3797877 Nov 02 23:09:57.208053 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:57.207981 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" event={"ID":"7ef6e83d-2b3b-44c2-a973-86f2587ac616","Type":"ContainerStarted","Data":"de9ff54c6f30c6c8ce4ebf978b30b1e69a2db6aa3ab286efb5aca880fad54c2b"} Nov 02 23:09:57.208053 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:57.208030 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" event={"ID":"7ef6e83d-2b3b-44c2-a973-86f2587ac616","Type":"ContainerStarted","Data":"21fd918261c4a9807af472c1616f3e8baa2d95eca9b0fc0078a13f8bc3797877"} Nov 02 23:09:57.335106 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:57.335046 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" podStartSLOduration=1.335032854 podStartE2EDuration="1.335032854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:09:57.334447574 +0000 UTC m=+1283.572945256" watchObservedRunningTime="2026-11-02 23:09:57.335032854 +0000 UTC m=+1283.573530516" Nov 02 23:09:57.580155 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:57.576238 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" Nov 02 23:09:57.773641 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:57.773544 5727 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 23:09:57.773905 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:57.773875 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" podUID="61ae60a1-87a0-42bf-9342-0420cd1cbce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 23:09:57.846115 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:09:57.846083 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:09:58.772815 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:58.772777 5727 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 23:09:58.773018 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:09:58.772835 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" podUID="61ae60a1-87a0-42bf-9342-0420cd1cbce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 23:10:00.173656 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:00.173629 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:10:00.174105 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:00.174080 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:10:00.175505 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:00.175235 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 02 23:10:03.662974 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:03.662936 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm"] Nov 02 23:10:03.668285 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:03.668264 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:03.697218 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:03.697188 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:03.697360 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:03.697266 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40350d65-8ea1-4d78-882c-f7b75c61dccd-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:03.697438 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:03.697386 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40350d65-8ea1-4d78-882c-f7b75c61dccd-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:03.697530 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:03.697508 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:03.697668 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:03.697569 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40350d65-8ea1-4d78-882c-f7b75c61dccd-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:03.697668 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:03.697626 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:03.697668 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:03.697650 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:03.697839 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:03.697672 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:03.697839 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:03.697782 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nsbr\" (UniqueName: \"kubernetes.io/projected/40350d65-8ea1-4d78-882c-f7b75c61dccd-kube-api-access-2nsbr\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:03.799026 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:03.798990 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:03.799227 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:03.799080 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40350d65-8ea1-4d78-882c-f7b75c61dccd-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:03.799227 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:03.799121 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40350d65-8ea1-4d78-882c-f7b75c61dccd-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:03.799377 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:03.799241 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:03.799377 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:03.799267 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40350d65-8ea1-4d78-882c-f7b75c61dccd-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:03.799377 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:03.799298 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:03.799377 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:03.799332 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:03.799377 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:03.799369 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:03.799608 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:03.799434 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nsbr\" (UniqueName: \"kubernetes.io/projected/40350d65-8ea1-4d78-882c-f7b75c61dccd-kube-api-access-2nsbr\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:03.799608 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:03.799457 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:03.799694 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:03.799652 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:03.799842 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:03.799778 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:03.799978 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:03.799956 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40350d65-8ea1-4d78-882c-f7b75c61dccd-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:03.799978 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:03.799966 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:03.801751 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:03.801726 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:03.802121 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:03.802103 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40350d65-8ea1-4d78-882c-f7b75c61dccd-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:03.859863 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:03.859823 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40350d65-8ea1-4d78-882c-f7b75c61dccd-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:03.860030 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:03.859980 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nsbr\" (UniqueName: \"kubernetes.io/projected/40350d65-8ea1-4d78-882c-f7b75c61dccd-kube-api-access-2nsbr\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:03.924857 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:03.924766 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm"] Nov 02 23:10:03.986007 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:03.985973 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:04.426669 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:10:04.426639 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40350d65_8ea1_4d78_882c_f7b75c61dccd.slice/crio-98bed883dde4e655b622ec3bfdafb3916b1687aa661f9591ca865e813877afc4 WatchSource:0}: Error finding container 98bed883dde4e655b622ec3bfdafb3916b1687aa661f9591ca865e813877afc4: Status 404 returned error can't find the container with id 98bed883dde4e655b622ec3bfdafb3916b1687aa661f9591ca865e813877afc4 Nov 02 23:10:04.427281 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:04.427254 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm"] Nov 02 23:10:04.429721 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:04.429691 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:10:04.429830 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:04.429755 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:10:04.429830 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:04.429792 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:10:05.132016 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.131980 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m"] Nov 02 23:10:05.136376 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.136357 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:05.220043 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.220003 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe18fd85-9a72-4868-8763-038ea4a0a54a-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:05.220250 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.220079 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe18fd85-9a72-4868-8763-038ea4a0a54a-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:05.220250 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.220119 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:05.220730 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.220705 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:05.221471 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.221446 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:05.221593 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.221484 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pjsd\" (UniqueName: \"kubernetes.io/projected/fe18fd85-9a72-4868-8763-038ea4a0a54a-kube-api-access-7pjsd\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:05.221593 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.221555 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:05.221722 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.221697 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:05.221781 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.221735 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe18fd85-9a72-4868-8763-038ea4a0a54a-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:05.228920 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.228308 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m"] Nov 02 23:10:05.296261 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.296224 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" event={"ID":"40350d65-8ea1-4d78-882c-f7b75c61dccd","Type":"ContainerStarted","Data":"b283b41806dadd842746ea010d794d131af68310ac330487f0ae237624d9eaf1"} Nov 02 23:10:05.296528 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.296285 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" event={"ID":"40350d65-8ea1-4d78-882c-f7b75c61dccd","Type":"ContainerStarted","Data":"98bed883dde4e655b622ec3bfdafb3916b1687aa661f9591ca865e813877afc4"} Nov 02 23:10:05.323381 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.323344 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe18fd85-9a72-4868-8763-038ea4a0a54a-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:05.323554 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.323393 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe18fd85-9a72-4868-8763-038ea4a0a54a-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:05.323554 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.323426 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:05.323554 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.323549 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:05.323724 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.323628 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:05.323724 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.323660 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pjsd\" (UniqueName: \"kubernetes.io/projected/fe18fd85-9a72-4868-8763-038ea4a0a54a-kube-api-access-7pjsd\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:05.323824 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.323761 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:05.323879 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.323868 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:05.323933 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.323893 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe18fd85-9a72-4868-8763-038ea4a0a54a-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:05.324056 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.324030 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:05.324157 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.324105 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:05.324221 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.324173 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe18fd85-9a72-4868-8763-038ea4a0a54a-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:05.324381 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.324359 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:05.324533 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.324504 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:05.326473 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.326449 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:05.326747 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.326728 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe18fd85-9a72-4868-8763-038ea4a0a54a-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:05.389170 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.388950 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pjsd\" (UniqueName: \"kubernetes.io/projected/fe18fd85-9a72-4868-8763-038ea4a0a54a-kube-api-access-7pjsd\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:05.390854 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.390823 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe18fd85-9a72-4868-8763-038ea4a0a54a-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:05.416160 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.415535 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" podStartSLOduration=2.415516871 podStartE2EDuration="2.415516871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:10:05.414589236 +0000 UTC m=+1291.653086932" watchObservedRunningTime="2026-11-02 23:10:05.415516871 +0000 UTC m=+1291.654014534" Nov 02 23:10:05.462174 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.460385 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:05.935788 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.935759 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m"] Nov 02 23:10:05.936470 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:10:05.936443 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe18fd85_9a72_4868_8763_038ea4a0a54a.slice/crio-12cf1ebf8bcae65815763df6ebb73791f55166aa9ca8c0175b4c46692ad8e356 WatchSource:0}: Error finding container 12cf1ebf8bcae65815763df6ebb73791f55166aa9ca8c0175b4c46692ad8e356: Status 404 returned error can't find the container with id 12cf1ebf8bcae65815763df6ebb73791f55166aa9ca8c0175b4c46692ad8e356 Nov 02 23:10:05.944544 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.944510 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:10:05.944638 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.944576 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:10:05.944638 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.944605 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:10:05.987096 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.986364 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:05.988898 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.988870 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 23:10:05.989003 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:05.988930 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" podUID="40350d65-8ea1-4d78-882c-f7b75c61dccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 23:10:06.320727 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:06.318758 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" event={"ID":"fe18fd85-9a72-4868-8763-038ea4a0a54a","Type":"ContainerStarted","Data":"3941125a1b0a69e35c41042d94b10ef0841b5daec3ef5e9fde4d821d4c6c9bae"} Nov 02 23:10:06.320727 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:06.318820 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" event={"ID":"fe18fd85-9a72-4868-8763-038ea4a0a54a","Type":"ContainerStarted","Data":"12cf1ebf8bcae65815763df6ebb73791f55166aa9ca8c0175b4c46692ad8e356"} Nov 02 23:10:06.445140 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:06.445050 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" podStartSLOduration=1.4450311820000001 podStartE2EDuration="1.445031182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:10:06.440559687 +0000 UTC m=+1292.679057351" watchObservedRunningTime="2026-11-02 23:10:06.445031182 +0000 UTC m=+1292.683528847" Nov 02 23:10:06.470154 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:06.464204 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:06.589939 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:06.589858 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" Nov 02 23:10:06.878222 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:06.877863 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" Nov 02 23:10:06.996445 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:06.989523 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 23:10:06.996445 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:06.989588 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" podUID="40350d65-8ea1-4d78-882c-f7b75c61dccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 23:10:07.484154 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:07.467629 5727 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 02 23:10:07.484154 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:07.467706 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" podUID="fe18fd85-9a72-4868-8763-038ea4a0a54a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 02 23:10:07.897153 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:07.896290 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" podUID="d3799f21-d78b-468d-9347-ca3f0968409d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:08.002837 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:08.001396 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 23:10:08.002837 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:08.001457 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" podUID="40350d65-8ea1-4d78-882c-f7b75c61dccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 23:10:08.035063 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:08.034238 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" podUID="a9048ee3-8ca0-4723-9009-f6033427b829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Nov 02 23:10:08.277529 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:08.275654 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" podUID="f83c8651-5db1-4e80-b9fb-ec5b793e7e94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:08.427306 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:08.427259 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" podUID="7ef6e83d-2b3b-44c2-a973-86f2587ac616" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:08.465703 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:08.465668 5727 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 02 23:10:08.465964 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:08.465937 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" podUID="fe18fd85-9a72-4868-8763-038ea4a0a54a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 02 23:10:08.896691 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:08.896641 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" podUID="d3799f21-d78b-468d-9347-ca3f0968409d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:08.900483 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:08.898519 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" Nov 02 23:10:08.988589 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:08.988554 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 23:10:08.988881 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:08.988846 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" podUID="40350d65-8ea1-4d78-882c-f7b75c61dccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 23:10:09.040266 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:09.040223 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" podUID="a9048ee3-8ca0-4723-9009-f6033427b829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Nov 02 23:10:09.040892 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:09.040871 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" Nov 02 23:10:09.076189 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:09.068158 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" Nov 02 23:10:09.299529 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:09.297231 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" podUID="f83c8651-5db1-4e80-b9fb-ec5b793e7e94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:09.299529 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:09.297968 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" Nov 02 23:10:09.626189 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:09.624517 5727 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 02 23:10:09.626189 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:09.624582 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" podUID="fe18fd85-9a72-4868-8763-038ea4a0a54a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 02 23:10:09.725104 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:09.725053 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" podUID="7ef6e83d-2b3b-44c2-a973-86f2587ac616" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 02 23:10:10.021359 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:10.021270 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 23:10:10.021939 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:10.021893 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" podUID="40350d65-8ea1-4d78-882c-f7b75c61dccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 23:10:10.057823 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:10.057767 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" podUID="a9048ee3-8ca0-4723-9009-f6033427b829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:10.265150 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:10.264242 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:10:10.265150 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:10.264328 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" podUID="575d202b-f514-4312-8dc7-dfad51c6ca19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:10.299209 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:10.298314 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" podUID="f83c8651-5db1-4e80-b9fb-ec5b793e7e94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:10.510151 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:10.463619 5727 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 02 23:10:10.532151 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:10.529783 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" podUID="fe18fd85-9a72-4868-8763-038ea4a0a54a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 02 23:10:10.734469 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:10.733466 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" podUID="7ef6e83d-2b3b-44c2-a973-86f2587ac616" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:10.738865 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:10.738840 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" Nov 02 23:10:10.991876 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:10.991784 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 23:10:10.992120 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:10.992093 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" podUID="40350d65-8ea1-4d78-882c-f7b75c61dccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 23:10:11.079388 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:11.079334 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" podUID="a9048ee3-8ca0-4723-9009-f6033427b829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Nov 02 23:10:11.192722 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:11.189377 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" podUID="d3799f21-d78b-468d-9347-ca3f0968409d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:11.209950 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:11.209923 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" Nov 02 23:10:11.489162 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:11.488535 5727 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 02 23:10:11.489162 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:11.488601 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" podUID="fe18fd85-9a72-4868-8763-038ea4a0a54a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 02 23:10:11.533665 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:11.533633 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" Nov 02 23:10:12.001417 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:12.001382 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 23:10:12.015558 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:12.015464 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" podUID="40350d65-8ea1-4d78-882c-f7b75c61dccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 23:10:12.475233 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:12.462434 5727 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 02 23:10:12.475233 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:12.462496 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" podUID="fe18fd85-9a72-4868-8763-038ea4a0a54a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 02 23:10:12.992194 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:12.986846 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 23:10:12.992541 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:12.992503 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" podUID="40350d65-8ea1-4d78-882c-f7b75c61dccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 23:10:13.462471 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:13.462434 5727 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 02 23:10:13.462752 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:13.462726 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" podUID="fe18fd85-9a72-4868-8763-038ea4a0a54a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 02 23:10:14.466876 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:14.466842 5727 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 02 23:10:14.467762 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:14.467561 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" podUID="fe18fd85-9a72-4868-8763-038ea4a0a54a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 02 23:10:14.919981 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:14.919943 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:14.920429 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:14.920414 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:14.921511 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:14.921494 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:10:15.462588 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:15.462553 5727 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 02 23:10:15.462759 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:15.462620 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" podUID="fe18fd85-9a72-4868-8763-038ea4a0a54a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 02 23:10:16.534391 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:16.534354 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:16.535557 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:16.535260 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:16.536453 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:16.536430 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 02 23:10:27.846598 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:10:27.846551 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:10:32.940269 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:32.940220 5727 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" podUID="d3799f21-d78b-468d-9347-ca3f0968409d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/health\": context deadline exceeded" Nov 02 23:10:33.108190 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:33.091312 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" podUID="d3799f21-d78b-468d-9347-ca3f0968409d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:33.108190 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:33.101457 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" podUID="a9048ee3-8ca0-4723-9009-f6033427b829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:33.299458 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:33.299358 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" podUID="f83c8651-5db1-4e80-b9fb-ec5b793e7e94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:55.099757 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:55.099716 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" podUID="a9048ee3-8ca0-4723-9009-f6033427b829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:57.846473 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:10:57.846434 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:10:58.555593 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:58.555561 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-gfc2f"] Nov 02 23:10:58.560718 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:58.560690 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-gfc2f" Nov 02 23:10:58.617427 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:58.617396 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-gfc2f"] Nov 02 23:10:58.635177 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:58.627108 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd4d615f-189f-44b7-87cc-a7d64e9dfd81-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-gfc2f\" (UID: \"fd4d615f-189f-44b7-87cc-a7d64e9dfd81\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-gfc2f" Nov 02 23:10:58.635177 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:58.627315 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5twlt\" (UniqueName: \"kubernetes.io/projected/fd4d615f-189f-44b7-87cc-a7d64e9dfd81-kube-api-access-5twlt\") pod \"payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-gfc2f\" (UID: \"fd4d615f-189f-44b7-87cc-a7d64e9dfd81\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-gfc2f" Nov 02 23:10:58.662314 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:58.662282 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a7a67875-dockercfg-44n9v\"" Nov 02 23:10:58.662533 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:58.662280 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a7a67875\"" Nov 02 23:10:58.732233 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:58.731917 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5twlt\" (UniqueName: \"kubernetes.io/projected/fd4d615f-189f-44b7-87cc-a7d64e9dfd81-kube-api-access-5twlt\") pod \"payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-gfc2f\" (UID: \"fd4d615f-189f-44b7-87cc-a7d64e9dfd81\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-gfc2f" Nov 02 23:10:58.732233 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:58.732149 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd4d615f-189f-44b7-87cc-a7d64e9dfd81-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-gfc2f\" (UID: \"fd4d615f-189f-44b7-87cc-a7d64e9dfd81\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-gfc2f" Nov 02 23:10:58.733068 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:58.733037 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd4d615f-189f-44b7-87cc-a7d64e9dfd81-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-gfc2f\" (UID: \"fd4d615f-189f-44b7-87cc-a7d64e9dfd81\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-gfc2f" Nov 02 23:10:58.809839 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:58.809762 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5twlt\" (UniqueName: \"kubernetes.io/projected/fd4d615f-189f-44b7-87cc-a7d64e9dfd81-kube-api-access-5twlt\") pod \"payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-gfc2f\" (UID: \"fd4d615f-189f-44b7-87cc-a7d64e9dfd81\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-gfc2f" Nov 02 23:10:58.883300 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:58.883266 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-gfc2f" Nov 02 23:10:59.270288 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:59.270259 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-gfc2f"] Nov 02 23:10:59.271758 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:10:59.271724 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd4d615f_189f_44b7_87cc_a7d64e9dfd81.slice/crio-4331efecc38136c526249bb28fd73872f77aee730d5674aebe09a07591d5ebab WatchSource:0}: Error finding container 4331efecc38136c526249bb28fd73872f77aee730d5674aebe09a07591d5ebab: Status 404 returned error can't find the container with id 4331efecc38136c526249bb28fd73872f77aee730d5674aebe09a07591d5ebab Nov 02 23:10:59.398455 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:59.398419 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-gfc2f" event={"ID":"fd4d615f-189f-44b7-87cc-a7d64e9dfd81","Type":"ContainerStarted","Data":"85278bcda8cc8c37833df664ce38fa215093bd1fdfeb911c22148e64beb7bbb9"} Nov 02 23:10:59.398600 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:59.398480 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-gfc2f" event={"ID":"fd4d615f-189f-44b7-87cc-a7d64e9dfd81","Type":"ContainerStarted","Data":"4331efecc38136c526249bb28fd73872f77aee730d5674aebe09a07591d5ebab"} Nov 02 23:10:59.398600 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:59.398591 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-gfc2f" Nov 02 23:10:59.518330 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:10:59.518208 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-gfc2f" podStartSLOduration=1.5181921699999998 podStartE2EDuration="1.51819217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:10:59.51415448 +0000 UTC m=+1345.752652144" watchObservedRunningTime="2026-11-02 23:10:59.51819217 +0000 UTC m=+1345.756689832" Nov 02 23:11:10.409235 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:11:10.409209 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-gfc2f" Nov 02 23:11:27.846615 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:11:27.846581 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:11:28.051334 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:11:28.051305 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj"] Nov 02 23:11:28.055454 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:11:28.055431 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj" Nov 02 23:11:28.107225 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:11:28.107158 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a7a67875-serving-cert\"" Nov 02 23:11:28.160463 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:11:28.160431 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj"] Nov 02 23:11:28.170472 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:11:28.170444 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9be2be64-8a5a-4275-a973-1a78b432c62f-metrics-certs\") pod \"maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj\" (UID: \"9be2be64-8a5a-4275-a973-1a78b432c62f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj" Nov 02 23:11:28.170675 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:11:28.170656 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whjnm\" (UniqueName: \"kubernetes.io/projected/9be2be64-8a5a-4275-a973-1a78b432c62f-kube-api-access-whjnm\") pod \"maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj\" (UID: \"9be2be64-8a5a-4275-a973-1a78b432c62f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj" Nov 02 23:11:28.170750 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:11:28.170699 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9be2be64-8a5a-4275-a973-1a78b432c62f-maas-api-tls\") pod \"maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj\" (UID: \"9be2be64-8a5a-4275-a973-1a78b432c62f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj" Nov 02 23:11:28.170839 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:11:28.170821 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9be2be64-8a5a-4275-a973-1a78b432c62f-openshift-service-ca\") pod \"maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj\" (UID: \"9be2be64-8a5a-4275-a973-1a78b432c62f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj" Nov 02 23:11:28.272433 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:11:28.272403 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9be2be64-8a5a-4275-a973-1a78b432c62f-metrics-certs\") pod \"maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj\" (UID: \"9be2be64-8a5a-4275-a973-1a78b432c62f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj" Nov 02 23:11:28.272623 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:11:28.272532 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whjnm\" (UniqueName: \"kubernetes.io/projected/9be2be64-8a5a-4275-a973-1a78b432c62f-kube-api-access-whjnm\") pod \"maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj\" (UID: \"9be2be64-8a5a-4275-a973-1a78b432c62f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj" Nov 02 23:11:28.272623 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:11:28.272553 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9be2be64-8a5a-4275-a973-1a78b432c62f-maas-api-tls\") pod \"maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj\" (UID: \"9be2be64-8a5a-4275-a973-1a78b432c62f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj" Nov 02 23:11:28.272712 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:11:28.272630 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9be2be64-8a5a-4275-a973-1a78b432c62f-openshift-service-ca\") pod \"maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj\" (UID: \"9be2be64-8a5a-4275-a973-1a78b432c62f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj" Nov 02 23:11:28.272799 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:11:28.272776 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-serving-cert: secret "maas-api-e2e-mt-a7a67875-serving-cert" not found Nov 02 23:11:28.272862 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:11:28.272856 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9be2be64-8a5a-4275-a973-1a78b432c62f-maas-api-tls podName:9be2be64-8a5a-4275-a973-1a78b432c62f nodeName:}" failed. No retries permitted until 2026-11-02 23:11:28.772836151 +0000 UTC m=+1375.011333810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9be2be64-8a5a-4275-a973-1a78b432c62f-maas-api-tls") pod "maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj" (UID: "9be2be64-8a5a-4275-a973-1a78b432c62f") : secret "maas-api-e2e-mt-a7a67875-serving-cert" not found Nov 02 23:11:28.273254 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:11:28.273234 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9be2be64-8a5a-4275-a973-1a78b432c62f-openshift-service-ca\") pod \"maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj\" (UID: \"9be2be64-8a5a-4275-a973-1a78b432c62f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj" Nov 02 23:11:28.275026 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:11:28.275010 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9be2be64-8a5a-4275-a973-1a78b432c62f-metrics-certs\") pod \"maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj\" (UID: \"9be2be64-8a5a-4275-a973-1a78b432c62f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj" Nov 02 23:11:28.335840 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:11:28.335802 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whjnm\" (UniqueName: \"kubernetes.io/projected/9be2be64-8a5a-4275-a973-1a78b432c62f-kube-api-access-whjnm\") pod \"maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj\" (UID: \"9be2be64-8a5a-4275-a973-1a78b432c62f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj" Nov 02 23:11:28.778793 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:11:28.778754 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9be2be64-8a5a-4275-a973-1a78b432c62f-maas-api-tls\") pod \"maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj\" (UID: \"9be2be64-8a5a-4275-a973-1a78b432c62f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj" Nov 02 23:11:28.781580 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:11:28.781545 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9be2be64-8a5a-4275-a973-1a78b432c62f-maas-api-tls\") pod \"maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj\" (UID: \"9be2be64-8a5a-4275-a973-1a78b432c62f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj" Nov 02 23:11:28.970042 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:11:28.970013 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj" Nov 02 23:11:29.350896 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:11:29.350866 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj"] Nov 02 23:11:29.351553 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:11:29.351526 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9be2be64_8a5a_4275_a973_1a78b432c62f.slice/crio-d74f24c7c085356db06f509bfce35dfcd1e14874b741e651af8c06ae0fe6905d WatchSource:0}: Error finding container d74f24c7c085356db06f509bfce35dfcd1e14874b741e651af8c06ae0fe6905d: Status 404 returned error can't find the container with id d74f24c7c085356db06f509bfce35dfcd1e14874b741e651af8c06ae0fe6905d Nov 02 23:11:29.353026 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:11:29.353010 5727 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 23:11:29.757691 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:11:29.757602 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj" event={"ID":"9be2be64-8a5a-4275-a973-1a78b432c62f","Type":"ContainerStarted","Data":"d74f24c7c085356db06f509bfce35dfcd1e14874b741e651af8c06ae0fe6905d"} Nov 02 23:11:30.767461 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:11:30.767424 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj" event={"ID":"9be2be64-8a5a-4275-a973-1a78b432c62f","Type":"ContainerStarted","Data":"456d6f1876852d656d9ca1e8367c83086ed2d39afafa0ddd5bdd6fbe5d942b41"} Nov 02 23:11:30.767865 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:11:30.767496 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj" Nov 02 23:11:30.883868 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:11:30.883811 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj" podStartSLOduration=1.957860466 podStartE2EDuration="2.883797088s" totalImagesPullingTime="925.936622ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:11:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:11:30.883411285 +0000 UTC m=+1377.121908972" watchObservedRunningTime="2026-11-02 23:11:30.883797088 +0000 UTC m=+1377.122294747" Nov 02 23:11:36.780723 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:11:36.780695 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj" Nov 02 23:11:57.850104 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:11:57.850068 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:12:15.305254 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:15.305220 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8e34df-b48f777f9-t7tg2"] Nov 02 23:12:15.310853 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:15.310827 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8e34df-b48f777f9-t7tg2" Nov 02 23:12:15.364771 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:15.364739 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8e34df-b48f777f9-t7tg2"] Nov 02 23:12:15.415010 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:15.414978 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-8e34df-dockercfg-82tnr\"" Nov 02 23:12:15.415351 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:15.415333 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-8e34df\"" Nov 02 23:12:15.415595 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:15.415561 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/875c0021-b1a3-45b3-81a4-da889bf77c04-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8e34df-b48f777f9-t7tg2\" (UID: \"875c0021-b1a3-45b3-81a4-da889bf77c04\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8e34df-b48f777f9-t7tg2" Nov 02 23:12:15.415774 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:15.415752 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fjt6\" (UniqueName: \"kubernetes.io/projected/875c0021-b1a3-45b3-81a4-da889bf77c04-kube-api-access-7fjt6\") pod \"payload-pre-processing-e2e-api-a-8e34df-b48f777f9-t7tg2\" (UID: \"875c0021-b1a3-45b3-81a4-da889bf77c04\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8e34df-b48f777f9-t7tg2" Nov 02 23:12:15.519749 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:15.516907 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/875c0021-b1a3-45b3-81a4-da889bf77c04-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8e34df-b48f777f9-t7tg2\" (UID: \"875c0021-b1a3-45b3-81a4-da889bf77c04\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8e34df-b48f777f9-t7tg2" Nov 02 23:12:15.519749 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:15.517000 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fjt6\" (UniqueName: \"kubernetes.io/projected/875c0021-b1a3-45b3-81a4-da889bf77c04-kube-api-access-7fjt6\") pod \"payload-pre-processing-e2e-api-a-8e34df-b48f777f9-t7tg2\" (UID: \"875c0021-b1a3-45b3-81a4-da889bf77c04\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8e34df-b48f777f9-t7tg2" Nov 02 23:12:15.519749 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:15.517897 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/875c0021-b1a3-45b3-81a4-da889bf77c04-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8e34df-b48f777f9-t7tg2\" (UID: \"875c0021-b1a3-45b3-81a4-da889bf77c04\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8e34df-b48f777f9-t7tg2" Nov 02 23:12:15.578151 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:15.578092 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fjt6\" (UniqueName: \"kubernetes.io/projected/875c0021-b1a3-45b3-81a4-da889bf77c04-kube-api-access-7fjt6\") pod \"payload-pre-processing-e2e-api-a-8e34df-b48f777f9-t7tg2\" (UID: \"875c0021-b1a3-45b3-81a4-da889bf77c04\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8e34df-b48f777f9-t7tg2" Nov 02 23:12:15.636307 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:15.636274 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8e34df-b48f777f9-t7tg2" Nov 02 23:12:16.025824 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:16.025792 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8e34df-b48f777f9-t7tg2"] Nov 02 23:12:16.026391 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:12:16.026365 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod875c0021_b1a3_45b3_81a4_da889bf77c04.slice/crio-8c9bfa10c649ee5c2d4df55fa4527d7c337e5aa959515e87245330c86abb1e63 WatchSource:0}: Error finding container 8c9bfa10c649ee5c2d4df55fa4527d7c337e5aa959515e87245330c86abb1e63: Status 404 returned error can't find the container with id 8c9bfa10c649ee5c2d4df55fa4527d7c337e5aa959515e87245330c86abb1e63 Nov 02 23:12:16.254856 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:16.254810 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8e34df-b48f777f9-t7tg2" event={"ID":"875c0021-b1a3-45b3-81a4-da889bf77c04","Type":"ContainerStarted","Data":"19320e3a37ab578eea617d7d421768b17a3998f35567961db87f9dfe536894ec"} Nov 02 23:12:16.255032 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:16.254913 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8e34df-b48f777f9-t7tg2" event={"ID":"875c0021-b1a3-45b3-81a4-da889bf77c04","Type":"ContainerStarted","Data":"8c9bfa10c649ee5c2d4df55fa4527d7c337e5aa959515e87245330c86abb1e63"} Nov 02 23:12:16.255032 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:16.254943 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8e34df-b48f777f9-t7tg2" Nov 02 23:12:16.370685 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:16.370627 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8e34df-b48f777f9-t7tg2" podStartSLOduration=1.370613111 podStartE2EDuration="1.370613111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:16.36943533 +0000 UTC m=+1422.607933017" watchObservedRunningTime="2026-11-02 23:12:16.370613111 +0000 UTC m=+1422.609110773" Nov 02 23:12:25.976555 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:25.976506 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-745976967d-cjvjc"] Nov 02 23:12:25.995284 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:25.995242 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-745976967d-cjvjc" Nov 02 23:12:26.038617 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:26.038584 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-745976967d-cjvjc"] Nov 02 23:12:26.076149 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:26.074024 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qfhq\" (UniqueName: \"kubernetes.io/projected/8ec80997-938e-45b5-b52d-835acc2b2aec-kube-api-access-2qfhq\") pod \"payload-processing-external-model-745976967d-cjvjc\" (UID: \"8ec80997-938e-45b5-b52d-835acc2b2aec\") " pod="models-as-a-service/payload-processing-external-model-745976967d-cjvjc" Nov 02 23:12:26.076149 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:26.075162 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8ec80997-938e-45b5-b52d-835acc2b2aec-provider-credentials\") pod \"payload-processing-external-model-745976967d-cjvjc\" (UID: \"8ec80997-938e-45b5-b52d-835acc2b2aec\") " pod="models-as-a-service/payload-processing-external-model-745976967d-cjvjc" Nov 02 23:12:26.076149 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:26.075223 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ec80997-938e-45b5-b52d-835acc2b2aec-plugins-config-volume\") pod \"payload-processing-external-model-745976967d-cjvjc\" (UID: \"8ec80997-938e-45b5-b52d-835acc2b2aec\") " pod="models-as-a-service/payload-processing-external-model-745976967d-cjvjc" Nov 02 23:12:26.076149 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:26.075318 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8ec80997-938e-45b5-b52d-835acc2b2aec-routing-overlay-volume\") pod \"payload-processing-external-model-745976967d-cjvjc\" (UID: \"8ec80997-938e-45b5-b52d-835acc2b2aec\") " pod="models-as-a-service/payload-processing-external-model-745976967d-cjvjc" Nov 02 23:12:26.084272 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:26.084240 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-cf53d560958d\"" Nov 02 23:12:26.084431 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:26.084349 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 23:12:26.084431 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:26.084411 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 23:12:26.084543 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:26.084459 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 23:12:26.099184 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:26.099113 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 23:12:26.099322 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:26.099208 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-rqscl\"" Nov 02 23:12:26.176353 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:26.176317 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8ec80997-938e-45b5-b52d-835acc2b2aec-provider-credentials\") pod \"payload-processing-external-model-745976967d-cjvjc\" (UID: \"8ec80997-938e-45b5-b52d-835acc2b2aec\") " pod="models-as-a-service/payload-processing-external-model-745976967d-cjvjc" Nov 02 23:12:26.176552 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:26.176374 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ec80997-938e-45b5-b52d-835acc2b2aec-plugins-config-volume\") pod \"payload-processing-external-model-745976967d-cjvjc\" (UID: \"8ec80997-938e-45b5-b52d-835acc2b2aec\") " pod="models-as-a-service/payload-processing-external-model-745976967d-cjvjc" Nov 02 23:12:26.176552 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:26.176459 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8ec80997-938e-45b5-b52d-835acc2b2aec-routing-overlay-volume\") pod \"payload-processing-external-model-745976967d-cjvjc\" (UID: \"8ec80997-938e-45b5-b52d-835acc2b2aec\") " pod="models-as-a-service/payload-processing-external-model-745976967d-cjvjc" Nov 02 23:12:26.176552 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:26.176547 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qfhq\" (UniqueName: \"kubernetes.io/projected/8ec80997-938e-45b5-b52d-835acc2b2aec-kube-api-access-2qfhq\") pod \"payload-processing-external-model-745976967d-cjvjc\" (UID: \"8ec80997-938e-45b5-b52d-835acc2b2aec\") " pod="models-as-a-service/payload-processing-external-model-745976967d-cjvjc" Nov 02 23:12:26.177343 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:26.177304 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8ec80997-938e-45b5-b52d-835acc2b2aec-routing-overlay-volume\") pod \"payload-processing-external-model-745976967d-cjvjc\" (UID: \"8ec80997-938e-45b5-b52d-835acc2b2aec\") " pod="models-as-a-service/payload-processing-external-model-745976967d-cjvjc" Nov 02 23:12:26.177460 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:26.177344 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ec80997-938e-45b5-b52d-835acc2b2aec-plugins-config-volume\") pod \"payload-processing-external-model-745976967d-cjvjc\" (UID: \"8ec80997-938e-45b5-b52d-835acc2b2aec\") " pod="models-as-a-service/payload-processing-external-model-745976967d-cjvjc" Nov 02 23:12:26.179030 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:26.179006 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8ec80997-938e-45b5-b52d-835acc2b2aec-provider-credentials\") pod \"payload-processing-external-model-745976967d-cjvjc\" (UID: \"8ec80997-938e-45b5-b52d-835acc2b2aec\") " pod="models-as-a-service/payload-processing-external-model-745976967d-cjvjc" Nov 02 23:12:26.237631 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:26.237551 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qfhq\" (UniqueName: \"kubernetes.io/projected/8ec80997-938e-45b5-b52d-835acc2b2aec-kube-api-access-2qfhq\") pod \"payload-processing-external-model-745976967d-cjvjc\" (UID: \"8ec80997-938e-45b5-b52d-835acc2b2aec\") " pod="models-as-a-service/payload-processing-external-model-745976967d-cjvjc" Nov 02 23:12:26.310166 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:26.310120 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-745976967d-cjvjc" Nov 02 23:12:26.710187 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:26.710159 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-745976967d-cjvjc"] Nov 02 23:12:26.714485 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:12:26.712987 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec80997_938e_45b5_b52d_835acc2b2aec.slice/crio-05c44dc74daec872d8d9805458562e26a82586eb1de334968fecaf1f208b345b WatchSource:0}: Error finding container 05c44dc74daec872d8d9805458562e26a82586eb1de334968fecaf1f208b345b: Status 404 returned error can't find the container with id 05c44dc74daec872d8d9805458562e26a82586eb1de334968fecaf1f208b345b Nov 02 23:12:27.266262 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:27.266231 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8e34df-b48f777f9-t7tg2" Nov 02 23:12:27.423685 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:27.423655 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-745976967d-cjvjc" event={"ID":"8ec80997-938e-45b5-b52d-835acc2b2aec","Type":"ContainerStarted","Data":"028050e170e559b5d0757bc43fd2b9a1673ccc69f1482422df7fdae16f864c5b"} Nov 02 23:12:27.423879 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:27.423712 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-745976967d-cjvjc" event={"ID":"8ec80997-938e-45b5-b52d-835acc2b2aec","Type":"ContainerStarted","Data":"05c44dc74daec872d8d9805458562e26a82586eb1de334968fecaf1f208b345b"} Nov 02 23:12:27.423879 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:27.423797 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-745976967d-cjvjc" Nov 02 23:12:27.541215 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:27.541094 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-745976967d-cjvjc" podStartSLOduration=2.541078535 podStartE2EDuration="2.541078535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:27.539588911 +0000 UTC m=+1433.778086594" watchObservedRunningTime="2026-11-02 23:12:27.541078535 +0000 UTC m=+1433.779576196" Nov 02 23:12:27.847251 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:12:27.847216 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:12:38.435704 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:38.435654 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-745976967d-cjvjc" podUID="8ec80997-938e-45b5-b52d-835acc2b2aec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 23:12:38.647554 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:38.647513 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-745976967d-cjvjc"] Nov 02 23:12:38.649317 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:38.649281 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-745976967d-cjvjc" podUID="8ec80997-938e-45b5-b52d-835acc2b2aec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 23:12:43.648837 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:43.648736 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-745976967d-cjvjc" podUID="8ec80997-938e-45b5-b52d-835acc2b2aec" containerName="payload-processing" containerID="cri-o://028050e170e559b5d0757bc43fd2b9a1673ccc69f1482422df7fdae16f864c5b" gracePeriod=25 Nov 02 23:12:43.938074 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:43.938046 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-745976967d-cjvjc" Nov 02 23:12:43.954859 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:43.954822 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ec80997-938e-45b5-b52d-835acc2b2aec-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8ec80997-938e-45b5-b52d-835acc2b2aec-provider-credentials\") pod \"8ec80997-938e-45b5-b52d-835acc2b2aec\" (UID: \"8ec80997-938e-45b5-b52d-835acc2b2aec\") " Nov 02 23:12:43.955053 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:43.954956 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ec80997-938e-45b5-b52d-835acc2b2aec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ec80997-938e-45b5-b52d-835acc2b2aec-plugins-config-volume\") pod \"8ec80997-938e-45b5-b52d-835acc2b2aec\" (UID: \"8ec80997-938e-45b5-b52d-835acc2b2aec\") " Nov 02 23:12:43.955053 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:43.954993 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ec80997-938e-45b5-b52d-835acc2b2aec-kube-api-access-2qfhq\" (UniqueName: \"kubernetes.io/projected/8ec80997-938e-45b5-b52d-835acc2b2aec-kube-api-access-2qfhq\") pod \"8ec80997-938e-45b5-b52d-835acc2b2aec\" (UID: \"8ec80997-938e-45b5-b52d-835acc2b2aec\") " Nov 02 23:12:43.955192 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:43.955051 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ec80997-938e-45b5-b52d-835acc2b2aec-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8ec80997-938e-45b5-b52d-835acc2b2aec-routing-overlay-volume\") pod \"8ec80997-938e-45b5-b52d-835acc2b2aec\" (UID: \"8ec80997-938e-45b5-b52d-835acc2b2aec\") " Nov 02 23:12:43.955444 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:43.955415 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec80997-938e-45b5-b52d-835acc2b2aec-plugins-config-volume" pod "8ec80997-938e-45b5-b52d-835acc2b2aec" (UID: "8ec80997-938e-45b5-b52d-835acc2b2aec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:12:43.955539 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:43.955462 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec80997-938e-45b5-b52d-835acc2b2aec-routing-overlay-volume" pod "8ec80997-938e-45b5-b52d-835acc2b2aec" (UID: "8ec80997-938e-45b5-b52d-835acc2b2aec"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:12:43.955717 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:43.955697 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ec80997-938e-45b5-b52d-835acc2b2aec-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:12:43.955790 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:43.955722 5727 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8ec80997-938e-45b5-b52d-835acc2b2aec-routing-overlay-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:12:43.957421 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:43.957395 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec80997-938e-45b5-b52d-835acc2b2aec-provider-credentials" pod "8ec80997-938e-45b5-b52d-835acc2b2aec" (UID: "8ec80997-938e-45b5-b52d-835acc2b2aec"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:12:43.957513 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:43.957460 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec80997-938e-45b5-b52d-835acc2b2aec-kube-api-access-2qfhq" pod "8ec80997-938e-45b5-b52d-835acc2b2aec" (UID: "8ec80997-938e-45b5-b52d-835acc2b2aec"). InnerVolumeSpecName "kube-api-access-2qfhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:12:44.057653 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:44.057618 5727 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8ec80997-938e-45b5-b52d-835acc2b2aec-provider-credentials\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:12:44.057653 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:44.057656 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qfhq\" (UniqueName: \"kubernetes.io/projected/8ec80997-938e-45b5-b52d-835acc2b2aec-kube-api-access-2qfhq\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:12:44.560435 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:44.560392 5727 generic.go:356] "Generic (PLEG): container finished" podID="8ec80997-938e-45b5-b52d-835acc2b2aec" containerID="028050e170e559b5d0757bc43fd2b9a1673ccc69f1482422df7fdae16f864c5b" exitCode=0 Nov 02 23:12:44.560647 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:44.560438 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-745976967d-cjvjc" Nov 02 23:12:44.560647 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:44.560440 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-745976967d-cjvjc" event={"ID":"8ec80997-938e-45b5-b52d-835acc2b2aec","Type":"ContainerDied","Data":"028050e170e559b5d0757bc43fd2b9a1673ccc69f1482422df7fdae16f864c5b"} Nov 02 23:12:44.560647 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:44.560503 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-745976967d-cjvjc" event={"ID":"8ec80997-938e-45b5-b52d-835acc2b2aec","Type":"ContainerDied","Data":"05c44dc74daec872d8d9805458562e26a82586eb1de334968fecaf1f208b345b"} Nov 02 23:12:44.560647 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:44.560533 5727 scope.go:122] "RemoveContainer" containerID="028050e170e559b5d0757bc43fd2b9a1673ccc69f1482422df7fdae16f864c5b" Nov 02 23:12:44.572767 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:44.572753 5727 scope.go:122] "RemoveContainer" containerID="028050e170e559b5d0757bc43fd2b9a1673ccc69f1482422df7fdae16f864c5b" Nov 02 23:12:44.572986 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:12:44.572966 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"028050e170e559b5d0757bc43fd2b9a1673ccc69f1482422df7fdae16f864c5b\": container with ID starting with 028050e170e559b5d0757bc43fd2b9a1673ccc69f1482422df7fdae16f864c5b not found: ID does not exist" containerID="028050e170e559b5d0757bc43fd2b9a1673ccc69f1482422df7fdae16f864c5b" Nov 02 23:12:44.573049 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:44.572994 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"028050e170e559b5d0757bc43fd2b9a1673ccc69f1482422df7fdae16f864c5b"} err="failed to get container status \"028050e170e559b5d0757bc43fd2b9a1673ccc69f1482422df7fdae16f864c5b\": rpc error: code = NotFound desc = could not find container \"028050e170e559b5d0757bc43fd2b9a1673ccc69f1482422df7fdae16f864c5b\": container with ID starting with 028050e170e559b5d0757bc43fd2b9a1673ccc69f1482422df7fdae16f864c5b not found: ID does not exist" Nov 02 23:12:44.733447 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:44.733407 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-745976967d-cjvjc"] Nov 02 23:12:44.782989 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:44.782955 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-745976967d-cjvjc"] Nov 02 23:12:45.107110 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:45.107073 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k"] Nov 02 23:12:45.107914 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:45.107891 5727 cpu_manager.go:414] "containerMap: removing container" podUID="8ec80997-938e-45b5-b52d-835acc2b2aec" containerName="payload-processing" Nov 02 23:12:45.107914 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:45.107915 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ec80997-938e-45b5-b52d-835acc2b2aec" containerName="payload-processing" Nov 02 23:12:45.108182 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:45.108167 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ec80997-938e-45b5-b52d-835acc2b2aec" containerName="payload-processing" Nov 02 23:12:45.113238 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:45.113219 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k" Nov 02 23:12:45.165265 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:45.165232 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-8e34df-serving-cert\"" Nov 02 23:12:45.170288 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:45.170257 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k"] Nov 02 23:12:45.170942 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:45.170917 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzvlc\" (UniqueName: \"kubernetes.io/projected/af82c965-7e41-4e03-97fb-f84c11f0b73e-kube-api-access-qzvlc\") pod \"maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k\" (UID: \"af82c965-7e41-4e03-97fb-f84c11f0b73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k" Nov 02 23:12:45.171095 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:45.171076 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af82c965-7e41-4e03-97fb-f84c11f0b73e-metrics-certs\") pod \"maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k\" (UID: \"af82c965-7e41-4e03-97fb-f84c11f0b73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k" Nov 02 23:12:45.171252 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:45.171233 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af82c965-7e41-4e03-97fb-f84c11f0b73e-openshift-service-ca\") pod \"maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k\" (UID: \"af82c965-7e41-4e03-97fb-f84c11f0b73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k" Nov 02 23:12:45.171345 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:45.171330 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af82c965-7e41-4e03-97fb-f84c11f0b73e-maas-api-tls\") pod \"maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k\" (UID: \"af82c965-7e41-4e03-97fb-f84c11f0b73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k" Nov 02 23:12:45.273082 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:45.273047 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzvlc\" (UniqueName: \"kubernetes.io/projected/af82c965-7e41-4e03-97fb-f84c11f0b73e-kube-api-access-qzvlc\") pod \"maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k\" (UID: \"af82c965-7e41-4e03-97fb-f84c11f0b73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k" Nov 02 23:12:45.273279 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:45.273204 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af82c965-7e41-4e03-97fb-f84c11f0b73e-metrics-certs\") pod \"maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k\" (UID: \"af82c965-7e41-4e03-97fb-f84c11f0b73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k" Nov 02 23:12:45.273345 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:45.273327 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af82c965-7e41-4e03-97fb-f84c11f0b73e-openshift-service-ca\") pod \"maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k\" (UID: \"af82c965-7e41-4e03-97fb-f84c11f0b73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k" Nov 02 23:12:45.273442 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:45.273428 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af82c965-7e41-4e03-97fb-f84c11f0b73e-maas-api-tls\") pod \"maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k\" (UID: \"af82c965-7e41-4e03-97fb-f84c11f0b73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k" Nov 02 23:12:45.273567 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:12:45.273547 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-8e34df-serving-cert: secret "maas-api-e2e-api-a-8e34df-serving-cert" not found Nov 02 23:12:45.273643 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:12:45.273630 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af82c965-7e41-4e03-97fb-f84c11f0b73e-maas-api-tls podName:af82c965-7e41-4e03-97fb-f84c11f0b73e nodeName:}" failed. No retries permitted until 2026-11-02 23:12:45.773607226 +0000 UTC m=+1452.012104881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/af82c965-7e41-4e03-97fb-f84c11f0b73e-maas-api-tls") pod "maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k" (UID: "af82c965-7e41-4e03-97fb-f84c11f0b73e") : secret "maas-api-e2e-api-a-8e34df-serving-cert" not found Nov 02 23:12:45.274231 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:45.274120 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af82c965-7e41-4e03-97fb-f84c11f0b73e-openshift-service-ca\") pod \"maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k\" (UID: \"af82c965-7e41-4e03-97fb-f84c11f0b73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k" Nov 02 23:12:45.275833 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:45.275815 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af82c965-7e41-4e03-97fb-f84c11f0b73e-metrics-certs\") pod \"maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k\" (UID: \"af82c965-7e41-4e03-97fb-f84c11f0b73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k" Nov 02 23:12:45.334549 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:45.334519 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzvlc\" (UniqueName: \"kubernetes.io/projected/af82c965-7e41-4e03-97fb-f84c11f0b73e-kube-api-access-qzvlc\") pod \"maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k\" (UID: \"af82c965-7e41-4e03-97fb-f84c11f0b73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k" Nov 02 23:12:45.782783 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:45.782742 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af82c965-7e41-4e03-97fb-f84c11f0b73e-maas-api-tls\") pod \"maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k\" (UID: \"af82c965-7e41-4e03-97fb-f84c11f0b73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k" Nov 02 23:12:45.785408 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:45.785379 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af82c965-7e41-4e03-97fb-f84c11f0b73e-maas-api-tls\") pod \"maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k\" (UID: \"af82c965-7e41-4e03-97fb-f84c11f0b73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k" Nov 02 23:12:45.884850 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:45.884818 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ec80997-938e-45b5-b52d-835acc2b2aec" path="/var/lib/kubelet/pods/8ec80997-938e-45b5-b52d-835acc2b2aec/volumes" Nov 02 23:12:46.027507 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:46.027468 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k" Nov 02 23:12:46.411010 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:46.410979 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k"] Nov 02 23:12:46.412096 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:12:46.412065 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf82c965_7e41_4e03_97fb_f84c11f0b73e.slice/crio-b2fbf071466aa15bae0f45b64c9f04bd7d26e5dfd2c317afffc237f6c7422192 WatchSource:0}: Error finding container b2fbf071466aa15bae0f45b64c9f04bd7d26e5dfd2c317afffc237f6c7422192: Status 404 returned error can't find the container with id b2fbf071466aa15bae0f45b64c9f04bd7d26e5dfd2c317afffc237f6c7422192 Nov 02 23:12:46.578937 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:46.578900 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k" event={"ID":"af82c965-7e41-4e03-97fb-f84c11f0b73e","Type":"ContainerStarted","Data":"b2fbf071466aa15bae0f45b64c9f04bd7d26e5dfd2c317afffc237f6c7422192"} Nov 02 23:12:47.589502 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:47.589462 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k" event={"ID":"af82c965-7e41-4e03-97fb-f84c11f0b73e","Type":"ContainerStarted","Data":"39c96e08c90b15dcf5e15ada95a6adca8c046500e0cc5aaceb1191f47eb24342"} Nov 02 23:12:47.589927 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:47.589665 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k" Nov 02 23:12:47.706840 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:47.706785 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k" podStartSLOduration=1.794717138 podStartE2EDuration="2.706770815s" totalImagesPullingTime="912.053677ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:47.705601565 +0000 UTC m=+1453.944099248" watchObservedRunningTime="2026-11-02 23:12:47.706770815 +0000 UTC m=+1453.945268476" Nov 02 23:12:53.603612 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:53.603582 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k" Nov 02 23:12:56.315639 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:56.315608 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5153fe-67644d745-stq6l"] Nov 02 23:12:56.320058 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:56.320040 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5153fe-67644d745-stq6l" Nov 02 23:12:56.420637 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:56.420604 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5153fe-67644d745-stq6l"] Nov 02 23:12:56.422518 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:56.422484 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-5153fe-dockercfg-qggrn\"" Nov 02 23:12:56.422785 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:56.422768 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-5153fe\"" Nov 02 23:12:56.485794 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:56.485758 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd8121c0-31b6-40d5-be58-3ada3872408c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5153fe-67644d745-stq6l\" (UID: \"fd8121c0-31b6-40d5-be58-3ada3872408c\") " pod="openshift-ingress/payload-processing-e2e-api-a-5153fe-67644d745-stq6l" Nov 02 23:12:56.486432 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:56.486170 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg76k\" (UniqueName: \"kubernetes.io/projected/fd8121c0-31b6-40d5-be58-3ada3872408c-kube-api-access-vg76k\") pod \"payload-processing-e2e-api-a-5153fe-67644d745-stq6l\" (UID: \"fd8121c0-31b6-40d5-be58-3ada3872408c\") " pod="openshift-ingress/payload-processing-e2e-api-a-5153fe-67644d745-stq6l" Nov 02 23:12:56.588830 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:56.588733 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg76k\" (UniqueName: \"kubernetes.io/projected/fd8121c0-31b6-40d5-be58-3ada3872408c-kube-api-access-vg76k\") pod \"payload-processing-e2e-api-a-5153fe-67644d745-stq6l\" (UID: \"fd8121c0-31b6-40d5-be58-3ada3872408c\") " pod="openshift-ingress/payload-processing-e2e-api-a-5153fe-67644d745-stq6l" Nov 02 23:12:56.589007 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:56.588879 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd8121c0-31b6-40d5-be58-3ada3872408c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5153fe-67644d745-stq6l\" (UID: \"fd8121c0-31b6-40d5-be58-3ada3872408c\") " pod="openshift-ingress/payload-processing-e2e-api-a-5153fe-67644d745-stq6l" Nov 02 23:12:56.589602 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:56.589579 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd8121c0-31b6-40d5-be58-3ada3872408c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5153fe-67644d745-stq6l\" (UID: \"fd8121c0-31b6-40d5-be58-3ada3872408c\") " pod="openshift-ingress/payload-processing-e2e-api-a-5153fe-67644d745-stq6l" Nov 02 23:12:56.649685 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:56.649654 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg76k\" (UniqueName: \"kubernetes.io/projected/fd8121c0-31b6-40d5-be58-3ada3872408c-kube-api-access-vg76k\") pod \"payload-processing-e2e-api-a-5153fe-67644d745-stq6l\" (UID: \"fd8121c0-31b6-40d5-be58-3ada3872408c\") " pod="openshift-ingress/payload-processing-e2e-api-a-5153fe-67644d745-stq6l" Nov 02 23:12:56.938193 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:56.938092 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5153fe-67644d745-stq6l" Nov 02 23:12:57.321192 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:57.321165 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5153fe-67644d745-stq6l"] Nov 02 23:12:57.322161 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:12:57.322110 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd8121c0_31b6_40d5_be58_3ada3872408c.slice/crio-3bd43ea21577436b0e2c127ce5402a9ff1e355079d82feda54052d12ca019136 WatchSource:0}: Error finding container 3bd43ea21577436b0e2c127ce5402a9ff1e355079d82feda54052d12ca019136: Status 404 returned error can't find the container with id 3bd43ea21577436b0e2c127ce5402a9ff1e355079d82feda54052d12ca019136 Nov 02 23:12:57.686981 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:57.686871 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5153fe-67644d745-stq6l" event={"ID":"fd8121c0-31b6-40d5-be58-3ada3872408c","Type":"ContainerStarted","Data":"8e6c27c04988c37543a0b570bfea6103eec6552492d59de462eae620310f21e9"} Nov 02 23:12:57.686981 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:57.686944 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5153fe-67644d745-stq6l" event={"ID":"fd8121c0-31b6-40d5-be58-3ada3872408c","Type":"ContainerStarted","Data":"3bd43ea21577436b0e2c127ce5402a9ff1e355079d82feda54052d12ca019136"} Nov 02 23:12:57.686981 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:57.686977 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-5153fe-67644d745-stq6l" Nov 02 23:12:57.804905 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:12:57.804854 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-5153fe-67644d745-stq6l" podStartSLOduration=1.804839316 podStartE2EDuration="1.804839316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:57.801515224 +0000 UTC m=+1464.040012885" watchObservedRunningTime="2026-11-02 23:12:57.804839316 +0000 UTC m=+1464.043336977" Nov 02 23:12:57.847231 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:12:57.847196 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:13:08.702085 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:08.702056 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-5153fe-67644d745-stq6l" Nov 02 23:13:16.985212 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:16.985180 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t"] Nov 02 23:13:16.990714 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:16.990693 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" Nov 02 23:13:17.045614 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:17.045581 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t"] Nov 02 23:13:17.090668 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:17.090633 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ec2f9a\"/\"openshift-service-ca.crt\"" Nov 02 23:13:17.090668 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:17.090633 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ec2f9a\"/\"default-dockercfg-c8pcq\"" Nov 02 23:13:17.090668 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:17.090641 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ec2f9a\"/\"kube-root-ca.crt\"" Nov 02 23:13:17.092976 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:17.092954 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ec2f9a\"/\"distinct-w0-ec2f9a-kserve-self-signed-certs\"" Nov 02 23:13:17.124353 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:17.124316 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a46665a-4362-4a43-8320-7dedd086b8d5-home\") pod \"distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t\" (UID: \"9a46665a-4362-4a43-8320-7dedd086b8d5\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" Nov 02 23:13:17.124667 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:17.124631 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m7c9\" (UniqueName: \"kubernetes.io/projected/9a46665a-4362-4a43-8320-7dedd086b8d5-kube-api-access-9m7c9\") pod \"distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t\" (UID: \"9a46665a-4362-4a43-8320-7dedd086b8d5\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" Nov 02 23:13:17.124792 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:17.124746 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a46665a-4362-4a43-8320-7dedd086b8d5-dshm\") pod \"distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t\" (UID: \"9a46665a-4362-4a43-8320-7dedd086b8d5\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" Nov 02 23:13:17.127174 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:17.124909 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a46665a-4362-4a43-8320-7dedd086b8d5-tls-certs\") pod \"distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t\" (UID: \"9a46665a-4362-4a43-8320-7dedd086b8d5\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" Nov 02 23:13:17.127174 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:17.124977 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a46665a-4362-4a43-8320-7dedd086b8d5-model-cache\") pod \"distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t\" (UID: \"9a46665a-4362-4a43-8320-7dedd086b8d5\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" Nov 02 23:13:17.127174 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:17.125019 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a46665a-4362-4a43-8320-7dedd086b8d5-tmp-dir\") pod \"distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t\" (UID: \"9a46665a-4362-4a43-8320-7dedd086b8d5\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" Nov 02 23:13:17.225958 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:17.225921 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a46665a-4362-4a43-8320-7dedd086b8d5-tls-certs\") pod \"distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t\" (UID: \"9a46665a-4362-4a43-8320-7dedd086b8d5\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" Nov 02 23:13:17.226219 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:17.225978 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a46665a-4362-4a43-8320-7dedd086b8d5-model-cache\") pod \"distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t\" (UID: \"9a46665a-4362-4a43-8320-7dedd086b8d5\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" Nov 02 23:13:17.226219 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:17.226004 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a46665a-4362-4a43-8320-7dedd086b8d5-tmp-dir\") pod \"distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t\" (UID: \"9a46665a-4362-4a43-8320-7dedd086b8d5\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" Nov 02 23:13:17.226219 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:17.226067 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a46665a-4362-4a43-8320-7dedd086b8d5-home\") pod \"distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t\" (UID: \"9a46665a-4362-4a43-8320-7dedd086b8d5\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" Nov 02 23:13:17.226219 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:17.226101 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m7c9\" (UniqueName: \"kubernetes.io/projected/9a46665a-4362-4a43-8320-7dedd086b8d5-kube-api-access-9m7c9\") pod \"distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t\" (UID: \"9a46665a-4362-4a43-8320-7dedd086b8d5\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" Nov 02 23:13:17.226468 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:17.226236 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a46665a-4362-4a43-8320-7dedd086b8d5-dshm\") pod \"distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t\" (UID: \"9a46665a-4362-4a43-8320-7dedd086b8d5\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" Nov 02 23:13:17.226525 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:17.226497 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a46665a-4362-4a43-8320-7dedd086b8d5-tmp-dir\") pod \"distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t\" (UID: \"9a46665a-4362-4a43-8320-7dedd086b8d5\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" Nov 02 23:13:17.226588 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:17.226545 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a46665a-4362-4a43-8320-7dedd086b8d5-home\") pod \"distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t\" (UID: \"9a46665a-4362-4a43-8320-7dedd086b8d5\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" Nov 02 23:13:17.226637 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:17.226583 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a46665a-4362-4a43-8320-7dedd086b8d5-model-cache\") pod \"distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t\" (UID: \"9a46665a-4362-4a43-8320-7dedd086b8d5\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" Nov 02 23:13:17.228630 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:17.228608 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a46665a-4362-4a43-8320-7dedd086b8d5-dshm\") pod \"distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t\" (UID: \"9a46665a-4362-4a43-8320-7dedd086b8d5\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" Nov 02 23:13:17.229008 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:17.228989 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a46665a-4362-4a43-8320-7dedd086b8d5-tls-certs\") pod \"distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t\" (UID: \"9a46665a-4362-4a43-8320-7dedd086b8d5\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" Nov 02 23:13:17.287233 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:17.287162 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m7c9\" (UniqueName: \"kubernetes.io/projected/9a46665a-4362-4a43-8320-7dedd086b8d5-kube-api-access-9m7c9\") pod \"distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t\" (UID: \"9a46665a-4362-4a43-8320-7dedd086b8d5\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" Nov 02 23:13:17.306618 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:17.306586 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" Nov 02 23:13:17.700750 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:17.700719 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t"] Nov 02 23:13:17.701144 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:13:17.701100 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a46665a_4362_4a43_8320_7dedd086b8d5.slice/crio-736e2b6736892517bf5d523093a3cebb02a621ea48f8bfd36325c2a39c51f151 WatchSource:0}: Error finding container 736e2b6736892517bf5d523093a3cebb02a621ea48f8bfd36325c2a39c51f151: Status 404 returned error can't find the container with id 736e2b6736892517bf5d523093a3cebb02a621ea48f8bfd36325c2a39c51f151 Nov 02 23:13:17.919082 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:17.919044 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" event={"ID":"9a46665a-4362-4a43-8320-7dedd086b8d5","Type":"ContainerStarted","Data":"3bd2a74951b111dbb2369d5192c9c521c7268cd78f91af6ca43cb803cd68ec46"} Nov 02 23:13:17.919287 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:17.919101 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" event={"ID":"9a46665a-4362-4a43-8320-7dedd086b8d5","Type":"ContainerStarted","Data":"736e2b6736892517bf5d523093a3cebb02a621ea48f8bfd36325c2a39c51f151"} Nov 02 23:13:18.035380 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:18.035273 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" podStartSLOduration=2.035257324 podStartE2EDuration="2.035257324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:13:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:13:18.03383577 +0000 UTC m=+1484.272333459" watchObservedRunningTime="2026-11-02 23:13:18.035257324 +0000 UTC m=+1484.273754986" Nov 02 23:13:18.307090 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:18.307010 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" Nov 02 23:13:27.307231 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:27.307193 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" Nov 02 23:13:27.320734 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:27.320705 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" Nov 02 23:13:27.846212 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:13:27.846173 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:13:28.041317 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:28.041284 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" Nov 02 23:13:32.992649 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:32.992611 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2"] Nov 02 23:13:32.997345 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:32.997323 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2" Nov 02 23:13:33.100459 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:33.100425 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-5153fe-serving-cert\"" Nov 02 23:13:33.105935 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:33.105900 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2"] Nov 02 23:13:33.111258 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:33.111232 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnf6n\" (UniqueName: \"kubernetes.io/projected/e7f2af2d-71e7-42c8-a05e-86379dd05a44-kube-api-access-jnf6n\") pod \"maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2\" (UID: \"e7f2af2d-71e7-42c8-a05e-86379dd05a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2" Nov 02 23:13:33.111362 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:33.111343 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f2af2d-71e7-42c8-a05e-86379dd05a44-openshift-service-ca\") pod \"maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2\" (UID: \"e7f2af2d-71e7-42c8-a05e-86379dd05a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2" Nov 02 23:13:33.111412 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:33.111366 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7f2af2d-71e7-42c8-a05e-86379dd05a44-maas-api-tls\") pod \"maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2\" (UID: \"e7f2af2d-71e7-42c8-a05e-86379dd05a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2" Nov 02 23:13:33.111544 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:33.111523 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f2af2d-71e7-42c8-a05e-86379dd05a44-metrics-certs\") pod \"maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2\" (UID: \"e7f2af2d-71e7-42c8-a05e-86379dd05a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2" Nov 02 23:13:33.213089 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:33.213052 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f2af2d-71e7-42c8-a05e-86379dd05a44-openshift-service-ca\") pod \"maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2\" (UID: \"e7f2af2d-71e7-42c8-a05e-86379dd05a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2" Nov 02 23:13:33.213311 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:33.213116 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7f2af2d-71e7-42c8-a05e-86379dd05a44-maas-api-tls\") pod \"maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2\" (UID: \"e7f2af2d-71e7-42c8-a05e-86379dd05a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2" Nov 02 23:13:33.213373 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:33.213337 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f2af2d-71e7-42c8-a05e-86379dd05a44-metrics-certs\") pod \"maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2\" (UID: \"e7f2af2d-71e7-42c8-a05e-86379dd05a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2" Nov 02 23:13:33.213465 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:33.213432 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnf6n\" (UniqueName: \"kubernetes.io/projected/e7f2af2d-71e7-42c8-a05e-86379dd05a44-kube-api-access-jnf6n\") pod \"maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2\" (UID: \"e7f2af2d-71e7-42c8-a05e-86379dd05a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2" Nov 02 23:13:33.221148 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:33.214931 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f2af2d-71e7-42c8-a05e-86379dd05a44-openshift-service-ca\") pod \"maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2\" (UID: \"e7f2af2d-71e7-42c8-a05e-86379dd05a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2" Nov 02 23:13:33.224805 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:33.224778 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f2af2d-71e7-42c8-a05e-86379dd05a44-metrics-certs\") pod \"maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2\" (UID: \"e7f2af2d-71e7-42c8-a05e-86379dd05a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2" Nov 02 23:13:33.225028 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:33.225008 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7f2af2d-71e7-42c8-a05e-86379dd05a44-maas-api-tls\") pod \"maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2\" (UID: \"e7f2af2d-71e7-42c8-a05e-86379dd05a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2" Nov 02 23:13:33.275647 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:33.275566 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnf6n\" (UniqueName: \"kubernetes.io/projected/e7f2af2d-71e7-42c8-a05e-86379dd05a44-kube-api-access-jnf6n\") pod \"maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2\" (UID: \"e7f2af2d-71e7-42c8-a05e-86379dd05a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2" Nov 02 23:13:33.313591 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:33.313554 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2" Nov 02 23:13:33.733661 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:13:33.733630 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f2af2d_71e7_42c8_a05e_86379dd05a44.slice/crio-bc55501f082530f513b98bf940360399ff31bfe2d71e2772a8d0872ae509962c WatchSource:0}: Error finding container bc55501f082530f513b98bf940360399ff31bfe2d71e2772a8d0872ae509962c: Status 404 returned error can't find the container with id bc55501f082530f513b98bf940360399ff31bfe2d71e2772a8d0872ae509962c Nov 02 23:13:33.733837 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:33.733812 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2"] Nov 02 23:13:34.099815 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:34.099761 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2" event={"ID":"e7f2af2d-71e7-42c8-a05e-86379dd05a44","Type":"ContainerStarted","Data":"bc55501f082530f513b98bf940360399ff31bfe2d71e2772a8d0872ae509962c"} Nov 02 23:13:34.342979 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:34.342950 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 02 23:13:34.355697 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:34.355615 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 02 23:13:34.358236 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:34.358210 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 23:13:34.371187 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:34.371163 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 23:13:35.110410 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:35.110373 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2" event={"ID":"e7f2af2d-71e7-42c8-a05e-86379dd05a44","Type":"ContainerStarted","Data":"fa0f61ac49d30da31737161fcc9c29fe441f0b176c6d0964a3bf722cc36452b7"} Nov 02 23:13:35.110839 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:35.110480 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2" Nov 02 23:13:35.237206 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:35.237143 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2" podStartSLOduration=2.416015888 podStartE2EDuration="3.23711316s" totalImagesPullingTime="821.097272ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:13:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:13:35.234782947 +0000 UTC m=+1501.473280608" watchObservedRunningTime="2026-11-02 23:13:35.23711316 +0000 UTC m=+1501.475610822" Nov 02 23:13:41.124533 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:41.124506 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2" Nov 02 23:13:56.023274 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:56.023242 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r"] Nov 02 23:13:56.027562 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:56.027544 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" Nov 02 23:13:56.130881 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:56.130849 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ec2f9a\"/\"distinct-2-w0-ec2f9a-kserve-self-signed-certs\"" Nov 02 23:13:56.134761 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:56.134742 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r"] Nov 02 23:13:56.184371 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:56.182706 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nnb4\" (UniqueName: \"kubernetes.io/projected/cdc40e13-d236-4739-a4f5-098aea3fb308-kube-api-access-7nnb4\") pod \"distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r\" (UID: \"cdc40e13-d236-4739-a4f5-098aea3fb308\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" Nov 02 23:13:56.184371 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:56.182782 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc40e13-d236-4739-a4f5-098aea3fb308-tmp-dir\") pod \"distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r\" (UID: \"cdc40e13-d236-4739-a4f5-098aea3fb308\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" Nov 02 23:13:56.184371 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:56.182915 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc40e13-d236-4739-a4f5-098aea3fb308-tls-certs\") pod \"distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r\" (UID: \"cdc40e13-d236-4739-a4f5-098aea3fb308\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" Nov 02 23:13:56.184371 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:56.182962 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc40e13-d236-4739-a4f5-098aea3fb308-model-cache\") pod \"distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r\" (UID: \"cdc40e13-d236-4739-a4f5-098aea3fb308\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" Nov 02 23:13:56.184371 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:56.182995 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc40e13-d236-4739-a4f5-098aea3fb308-home\") pod \"distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r\" (UID: \"cdc40e13-d236-4739-a4f5-098aea3fb308\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" Nov 02 23:13:56.184371 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:56.183170 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc40e13-d236-4739-a4f5-098aea3fb308-dshm\") pod \"distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r\" (UID: \"cdc40e13-d236-4739-a4f5-098aea3fb308\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" Nov 02 23:13:56.284322 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:56.284222 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc40e13-d236-4739-a4f5-098aea3fb308-dshm\") pod \"distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r\" (UID: \"cdc40e13-d236-4739-a4f5-098aea3fb308\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" Nov 02 23:13:56.284502 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:56.284347 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nnb4\" (UniqueName: \"kubernetes.io/projected/cdc40e13-d236-4739-a4f5-098aea3fb308-kube-api-access-7nnb4\") pod \"distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r\" (UID: \"cdc40e13-d236-4739-a4f5-098aea3fb308\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" Nov 02 23:13:56.284502 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:56.284384 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc40e13-d236-4739-a4f5-098aea3fb308-tmp-dir\") pod \"distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r\" (UID: \"cdc40e13-d236-4739-a4f5-098aea3fb308\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" Nov 02 23:13:56.284502 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:56.284489 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc40e13-d236-4739-a4f5-098aea3fb308-tls-certs\") pod \"distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r\" (UID: \"cdc40e13-d236-4739-a4f5-098aea3fb308\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" Nov 02 23:13:56.284671 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:56.284529 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc40e13-d236-4739-a4f5-098aea3fb308-model-cache\") pod \"distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r\" (UID: \"cdc40e13-d236-4739-a4f5-098aea3fb308\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" Nov 02 23:13:56.284671 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:56.284560 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc40e13-d236-4739-a4f5-098aea3fb308-home\") pod \"distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r\" (UID: \"cdc40e13-d236-4739-a4f5-098aea3fb308\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" Nov 02 23:13:56.284840 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:56.284813 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc40e13-d236-4739-a4f5-098aea3fb308-tmp-dir\") pod \"distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r\" (UID: \"cdc40e13-d236-4739-a4f5-098aea3fb308\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" Nov 02 23:13:56.284896 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:56.284838 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc40e13-d236-4739-a4f5-098aea3fb308-model-cache\") pod \"distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r\" (UID: \"cdc40e13-d236-4739-a4f5-098aea3fb308\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" Nov 02 23:13:56.284944 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:56.284924 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc40e13-d236-4739-a4f5-098aea3fb308-home\") pod \"distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r\" (UID: \"cdc40e13-d236-4739-a4f5-098aea3fb308\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" Nov 02 23:13:56.286683 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:56.286657 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc40e13-d236-4739-a4f5-098aea3fb308-dshm\") pod \"distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r\" (UID: \"cdc40e13-d236-4739-a4f5-098aea3fb308\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" Nov 02 23:13:56.287002 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:56.286984 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc40e13-d236-4739-a4f5-098aea3fb308-tls-certs\") pod \"distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r\" (UID: \"cdc40e13-d236-4739-a4f5-098aea3fb308\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" Nov 02 23:13:56.345360 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:56.345327 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nnb4\" (UniqueName: \"kubernetes.io/projected/cdc40e13-d236-4739-a4f5-098aea3fb308-kube-api-access-7nnb4\") pod \"distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r\" (UID: \"cdc40e13-d236-4739-a4f5-098aea3fb308\") " pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" Nov 02 23:13:56.641882 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:56.641852 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" Nov 02 23:13:57.029110 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:57.029077 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r"] Nov 02 23:13:57.029458 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:13:57.029342 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdc40e13_d236_4739_a4f5_098aea3fb308.slice/crio-847d0e11b487c5e02f28c73ec1b425137257f4c6962fa8bc8c69906b6e471740 WatchSource:0}: Error finding container 847d0e11b487c5e02f28c73ec1b425137257f4c6962fa8bc8c69906b6e471740: Status 404 returned error can't find the container with id 847d0e11b487c5e02f28c73ec1b425137257f4c6962fa8bc8c69906b6e471740 Nov 02 23:13:57.293356 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:57.293283 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" event={"ID":"cdc40e13-d236-4739-a4f5-098aea3fb308","Type":"ContainerStarted","Data":"caae0e836faa6edc9ef88cb87cbeddf92a2f10aec95d2fa6ed3ae9b7b65c4d28"} Nov 02 23:13:57.293356 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:57.293321 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" event={"ID":"cdc40e13-d236-4739-a4f5-098aea3fb308","Type":"ContainerStarted","Data":"847d0e11b487c5e02f28c73ec1b425137257f4c6962fa8bc8c69906b6e471740"} Nov 02 23:13:57.411038 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:57.410969 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" podStartSLOduration=2.410951834 podStartE2EDuration="2.410951834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:13:57.409433259 +0000 UTC m=+1523.647930940" watchObservedRunningTime="2026-11-02 23:13:57.410951834 +0000 UTC m=+1523.649449509" Nov 02 23:13:57.642464 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:13:57.642427 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" Nov 02 23:13:57.849381 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:13:57.849345 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:14:06.642981 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:06.642943 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" Nov 02 23:14:06.657792 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:06.657768 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" Nov 02 23:14:07.396041 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:07.396006 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" Nov 02 23:14:10.097742 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:10.097708 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7705-cc6e6d"] Nov 02 23:14:10.103901 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:10.103877 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-cc6e6d" Nov 02 23:14:10.158214 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:10.158183 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-cc6e6d"] Nov 02 23:14:10.184984 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:10.184954 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nn5c\" (UniqueName: \"kubernetes.io/projected/86e6369f-29f7-4765-a9a3-bda42fafc746-kube-api-access-2nn5c\") pod \"test-curl-7705-cc6e6d\" (UID: \"86e6369f-29f7-4765-a9a3-bda42fafc746\") " pod="opendatahub/test-curl-7705-cc6e6d" Nov 02 23:14:10.202792 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:10.202763 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-b8v4g\"" Nov 02 23:14:10.286105 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:10.286073 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nn5c\" (UniqueName: \"kubernetes.io/projected/86e6369f-29f7-4765-a9a3-bda42fafc746-kube-api-access-2nn5c\") pod \"test-curl-7705-cc6e6d\" (UID: \"86e6369f-29f7-4765-a9a3-bda42fafc746\") " pod="opendatahub/test-curl-7705-cc6e6d" Nov 02 23:14:10.346745 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:10.346719 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nn5c\" (UniqueName: \"kubernetes.io/projected/86e6369f-29f7-4765-a9a3-bda42fafc746-kube-api-access-2nn5c\") pod \"test-curl-7705-cc6e6d\" (UID: \"86e6369f-29f7-4765-a9a3-bda42fafc746\") " pod="opendatahub/test-curl-7705-cc6e6d" Nov 02 23:14:10.420778 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:10.420703 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-cc6e6d" Nov 02 23:14:10.797780 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:10.797750 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-cc6e6d"] Nov 02 23:14:11.434411 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:11.434366 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-cc6e6d" event={"ID":"86e6369f-29f7-4765-a9a3-bda42fafc746","Type":"ContainerStarted","Data":"a6d24ba7fed4433ddcb9985c8b1b0f1e2d789ee9623e42f8544f5e38cd3136d3"} Nov 02 23:14:14.462178 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:14.462139 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-cc6e6d" event={"ID":"86e6369f-29f7-4765-a9a3-bda42fafc746","Type":"ContainerStarted","Data":"3c2c0cbe95155ee6b60ab84dcbf489e7a29774c819d8ef590504f39dc4272e8e"} Nov 02 23:14:14.575745 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:14.575697 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7705-cc6e6d" podStartSLOduration=1.690583632 podStartE2EDuration="4.575684712s" totalImagesPullingTime="2.88510108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:14:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:14:14.574864344 +0000 UTC m=+1540.813362006" watchObservedRunningTime="2026-11-02 23:14:14.575684712 +0000 UTC m=+1540.814182375" Nov 02 23:14:16.045754 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:16.045724 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7705-cc6e6d"] Nov 02 23:14:16.050054 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:16.050022 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7705-cc6e6d"] Nov 02 23:14:16.476532 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:16.476493 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7705-cc6e6d" podUID="86e6369f-29f7-4765-a9a3-bda42fafc746" containerName="test-curl-7705-cc6e6d" containerID="cri-o://3c2c0cbe95155ee6b60ab84dcbf489e7a29774c819d8ef590504f39dc4272e8e" gracePeriod=2 Nov 02 23:14:16.527747 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:14:16.527710 5727 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7705-cc6e6d\" is forbidden: User \"system:node:ip-10-0-135-176.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-176.us-west-1.compute.internal' and this object" podUID="86e6369f-29f7-4765-a9a3-bda42fafc746" pod="opendatahub/test-curl-7705-cc6e6d" Nov 02 23:14:18.742064 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:18.742006 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-cc6e6d" Nov 02 23:14:18.801897 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:18.801869 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86e6369f-29f7-4765-a9a3-bda42fafc746-kube-api-access-2nn5c\" (UniqueName: \"kubernetes.io/projected/86e6369f-29f7-4765-a9a3-bda42fafc746-kube-api-access-2nn5c\") pod \"86e6369f-29f7-4765-a9a3-bda42fafc746\" (UID: \"86e6369f-29f7-4765-a9a3-bda42fafc746\") " Nov 02 23:14:18.804194 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:18.804167 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86e6369f-29f7-4765-a9a3-bda42fafc746-kube-api-access-2nn5c" pod "86e6369f-29f7-4765-a9a3-bda42fafc746" (UID: "86e6369f-29f7-4765-a9a3-bda42fafc746"). InnerVolumeSpecName "kube-api-access-2nn5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:14:18.903954 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:18.903918 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nn5c\" (UniqueName: \"kubernetes.io/projected/86e6369f-29f7-4765-a9a3-bda42fafc746-kube-api-access-2nn5c\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:14:19.503595 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:19.503555 5727 generic.go:356] "Generic (PLEG): container finished" podID="86e6369f-29f7-4765-a9a3-bda42fafc746" containerID="3c2c0cbe95155ee6b60ab84dcbf489e7a29774c819d8ef590504f39dc4272e8e" exitCode=137 Nov 02 23:14:19.503802 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:19.503612 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-cc6e6d" Nov 02 23:14:19.503802 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:19.503665 5727 scope.go:122] "RemoveContainer" containerID="3c2c0cbe95155ee6b60ab84dcbf489e7a29774c819d8ef590504f39dc4272e8e" Nov 02 23:14:19.514734 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:19.514715 5727 scope.go:122] "RemoveContainer" containerID="3c2c0cbe95155ee6b60ab84dcbf489e7a29774c819d8ef590504f39dc4272e8e" Nov 02 23:14:19.515055 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:14:19.515037 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c2c0cbe95155ee6b60ab84dcbf489e7a29774c819d8ef590504f39dc4272e8e\": container with ID starting with 3c2c0cbe95155ee6b60ab84dcbf489e7a29774c819d8ef590504f39dc4272e8e not found: ID does not exist" containerID="3c2c0cbe95155ee6b60ab84dcbf489e7a29774c819d8ef590504f39dc4272e8e" Nov 02 23:14:19.515120 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:19.515062 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c2c0cbe95155ee6b60ab84dcbf489e7a29774c819d8ef590504f39dc4272e8e"} err="failed to get container status \"3c2c0cbe95155ee6b60ab84dcbf489e7a29774c819d8ef590504f39dc4272e8e\": rpc error: code = NotFound desc = could not find container \"3c2c0cbe95155ee6b60ab84dcbf489e7a29774c819d8ef590504f39dc4272e8e\": container with ID starting with 3c2c0cbe95155ee6b60ab84dcbf489e7a29774c819d8ef590504f39dc4272e8e not found: ID does not exist" Nov 02 23:14:19.563134 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:14:19.563102 5727 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7705-cc6e6d\" is forbidden: User \"system:node:ip-10-0-135-176.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-176.us-west-1.compute.internal' and this object" podUID="86e6369f-29f7-4765-a9a3-bda42fafc746" pod="opendatahub/test-curl-7705-cc6e6d" Nov 02 23:14:19.884971 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:19.884937 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86e6369f-29f7-4765-a9a3-bda42fafc746" path="/var/lib/kubelet/pods/86e6369f-29f7-4765-a9a3-bda42fafc746/volumes" Nov 02 23:14:25.829867 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:25.829833 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8ab111-799db7457d-4c4nn"] Nov 02 23:14:25.830551 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:25.830535 5727 cpu_manager.go:414] "containerMap: removing container" podUID="86e6369f-29f7-4765-a9a3-bda42fafc746" containerName="test-curl-7705-cc6e6d" Nov 02 23:14:25.830601 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:25.830553 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86e6369f-29f7-4765-a9a3-bda42fafc746" containerName="test-curl-7705-cc6e6d" Nov 02 23:14:25.830765 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:25.830751 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="86e6369f-29f7-4765-a9a3-bda42fafc746" containerName="test-curl-7705-cc6e6d" Nov 02 23:14:25.834418 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:25.834400 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8ab111-799db7457d-4c4nn" Nov 02 23:14:25.887637 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:25.887577 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-8ab111-serving-cert\"" Nov 02 23:14:25.902383 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:25.902350 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt9xw\" (UniqueName: \"kubernetes.io/projected/9dce87c6-bbea-42af-8ac5-0655d4b4d352-kube-api-access-pt9xw\") pod \"maas-api-e2e-api-b-8ab111-799db7457d-4c4nn\" (UID: \"9dce87c6-bbea-42af-8ac5-0655d4b4d352\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8ab111-799db7457d-4c4nn" Nov 02 23:14:25.902609 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:25.902581 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dce87c6-bbea-42af-8ac5-0655d4b4d352-maas-api-tls\") pod \"maas-api-e2e-api-b-8ab111-799db7457d-4c4nn\" (UID: \"9dce87c6-bbea-42af-8ac5-0655d4b4d352\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8ab111-799db7457d-4c4nn" Nov 02 23:14:25.902768 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:25.902749 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dce87c6-bbea-42af-8ac5-0655d4b4d352-openshift-service-ca\") pod \"maas-api-e2e-api-b-8ab111-799db7457d-4c4nn\" (UID: \"9dce87c6-bbea-42af-8ac5-0655d4b4d352\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8ab111-799db7457d-4c4nn" Nov 02 23:14:25.902809 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:25.902788 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dce87c6-bbea-42af-8ac5-0655d4b4d352-metrics-certs\") pod \"maas-api-e2e-api-b-8ab111-799db7457d-4c4nn\" (UID: \"9dce87c6-bbea-42af-8ac5-0655d4b4d352\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8ab111-799db7457d-4c4nn" Nov 02 23:14:25.941140 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:25.941088 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8ab111-799db7457d-4c4nn"] Nov 02 23:14:26.014964 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:26.014932 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dce87c6-bbea-42af-8ac5-0655d4b4d352-maas-api-tls\") pod \"maas-api-e2e-api-b-8ab111-799db7457d-4c4nn\" (UID: \"9dce87c6-bbea-42af-8ac5-0655d4b4d352\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8ab111-799db7457d-4c4nn" Nov 02 23:14:26.015241 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:26.015056 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dce87c6-bbea-42af-8ac5-0655d4b4d352-openshift-service-ca\") pod \"maas-api-e2e-api-b-8ab111-799db7457d-4c4nn\" (UID: \"9dce87c6-bbea-42af-8ac5-0655d4b4d352\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8ab111-799db7457d-4c4nn" Nov 02 23:14:26.015241 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:26.015084 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dce87c6-bbea-42af-8ac5-0655d4b4d352-metrics-certs\") pod \"maas-api-e2e-api-b-8ab111-799db7457d-4c4nn\" (UID: \"9dce87c6-bbea-42af-8ac5-0655d4b4d352\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8ab111-799db7457d-4c4nn" Nov 02 23:14:26.015362 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:14:26.015286 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-8ab111-serving-cert: secret "maas-api-e2e-api-b-8ab111-serving-cert" not found Nov 02 23:14:26.015422 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:14:26.015370 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dce87c6-bbea-42af-8ac5-0655d4b4d352-maas-api-tls podName:9dce87c6-bbea-42af-8ac5-0655d4b4d352 nodeName:}" failed. No retries permitted until 2026-11-02 23:14:26.515345198 +0000 UTC m=+1552.753842858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9dce87c6-bbea-42af-8ac5-0655d4b4d352-maas-api-tls") pod "maas-api-e2e-api-b-8ab111-799db7457d-4c4nn" (UID: "9dce87c6-bbea-42af-8ac5-0655d4b4d352") : secret "maas-api-e2e-api-b-8ab111-serving-cert" not found Nov 02 23:14:26.015485 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:26.015455 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt9xw\" (UniqueName: \"kubernetes.io/projected/9dce87c6-bbea-42af-8ac5-0655d4b4d352-kube-api-access-pt9xw\") pod \"maas-api-e2e-api-b-8ab111-799db7457d-4c4nn\" (UID: \"9dce87c6-bbea-42af-8ac5-0655d4b4d352\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8ab111-799db7457d-4c4nn" Nov 02 23:14:26.016291 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:26.016266 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dce87c6-bbea-42af-8ac5-0655d4b4d352-openshift-service-ca\") pod \"maas-api-e2e-api-b-8ab111-799db7457d-4c4nn\" (UID: \"9dce87c6-bbea-42af-8ac5-0655d4b4d352\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8ab111-799db7457d-4c4nn" Nov 02 23:14:26.018422 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:26.018401 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dce87c6-bbea-42af-8ac5-0655d4b4d352-metrics-certs\") pod \"maas-api-e2e-api-b-8ab111-799db7457d-4c4nn\" (UID: \"9dce87c6-bbea-42af-8ac5-0655d4b4d352\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8ab111-799db7457d-4c4nn" Nov 02 23:14:26.078200 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:26.078173 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt9xw\" (UniqueName: \"kubernetes.io/projected/9dce87c6-bbea-42af-8ac5-0655d4b4d352-kube-api-access-pt9xw\") pod \"maas-api-e2e-api-b-8ab111-799db7457d-4c4nn\" (UID: \"9dce87c6-bbea-42af-8ac5-0655d4b4d352\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8ab111-799db7457d-4c4nn" Nov 02 23:14:26.521997 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:26.521965 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dce87c6-bbea-42af-8ac5-0655d4b4d352-maas-api-tls\") pod \"maas-api-e2e-api-b-8ab111-799db7457d-4c4nn\" (UID: \"9dce87c6-bbea-42af-8ac5-0655d4b4d352\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8ab111-799db7457d-4c4nn" Nov 02 23:14:26.524752 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:26.524722 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dce87c6-bbea-42af-8ac5-0655d4b4d352-maas-api-tls\") pod \"maas-api-e2e-api-b-8ab111-799db7457d-4c4nn\" (UID: \"9dce87c6-bbea-42af-8ac5-0655d4b4d352\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8ab111-799db7457d-4c4nn" Nov 02 23:14:26.748657 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:26.748624 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8ab111-799db7457d-4c4nn" Nov 02 23:14:27.131605 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:27.131577 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8ab111-799db7457d-4c4nn"] Nov 02 23:14:27.133938 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:14:27.133893 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dce87c6_bbea_42af_8ac5_0655d4b4d352.slice/crio-eacbe46430eb23ca98cf8faf4f49a77adcf6a630fccd4b5d171ed0a9eef56816 WatchSource:0}: Error finding container eacbe46430eb23ca98cf8faf4f49a77adcf6a630fccd4b5d171ed0a9eef56816: Status 404 returned error can't find the container with id eacbe46430eb23ca98cf8faf4f49a77adcf6a630fccd4b5d171ed0a9eef56816 Nov 02 23:14:27.564525 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:27.564493 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8ab111-799db7457d-4c4nn" event={"ID":"9dce87c6-bbea-42af-8ac5-0655d4b4d352","Type":"ContainerStarted","Data":"eacbe46430eb23ca98cf8faf4f49a77adcf6a630fccd4b5d171ed0a9eef56816"} Nov 02 23:14:27.850432 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:14:27.850349 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:14:28.574846 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:28.574810 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8ab111-799db7457d-4c4nn" event={"ID":"9dce87c6-bbea-42af-8ac5-0655d4b4d352","Type":"ContainerStarted","Data":"4fd356862564d8b155f7244d402c0459a662b7678c8482aabed5527428fe236d"} Nov 02 23:14:28.575332 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:28.574967 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8ab111-799db7457d-4c4nn" Nov 02 23:14:28.694067 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:28.694011 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8ab111-799db7457d-4c4nn" podStartSLOduration=2.846881209 podStartE2EDuration="3.693998008s" totalImagesPullingTime="847.116799ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:14:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:14:28.69165135 +0000 UTC m=+1554.930149029" watchObservedRunningTime="2026-11-02 23:14:28.693998008 +0000 UTC m=+1554.932495671" Nov 02 23:14:30.058900 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:30.058867 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-gfc2f"] Nov 02 23:14:34.589849 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:34.589821 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8ab111-799db7457d-4c4nn" Nov 02 23:14:35.062459 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:35.062370 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-gfc2f" podUID="fd4d615f-189f-44b7-87cc-a7d64e9dfd81" containerName="payload-pre-processing" containerID="cri-o://85278bcda8cc8c37833df664ce38fa215093bd1fdfeb911c22148e64beb7bbb9" gracePeriod=25 Nov 02 23:14:40.407606 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:40.407570 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-gfc2f" podUID="fd4d615f-189f-44b7-87cc-a7d64e9dfd81" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 23:14:43.754725 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:43.754666 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bf51d6-6db79f996c-6c4dz"] Nov 02 23:14:43.761407 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:43.761386 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf51d6-6db79f996c-6c4dz" Nov 02 23:14:43.811441 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:43.811416 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m552\" (UniqueName: \"kubernetes.io/projected/d0d6dc88-070f-4891-8e58-8adb3606cde0-kube-api-access-7m552\") pod \"payload-pre-processing-e2e-api-a-bf51d6-6db79f996c-6c4dz\" (UID: \"d0d6dc88-070f-4891-8e58-8adb3606cde0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf51d6-6db79f996c-6c4dz" Nov 02 23:14:43.811657 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:43.811630 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0d6dc88-070f-4891-8e58-8adb3606cde0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bf51d6-6db79f996c-6c4dz\" (UID: \"d0d6dc88-070f-4891-8e58-8adb3606cde0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf51d6-6db79f996c-6c4dz" Nov 02 23:14:43.816022 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:43.815999 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bf51d6-6db79f996c-6c4dz"] Nov 02 23:14:43.864196 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:43.860303 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-bf51d6-dockercfg-d8znb\"" Nov 02 23:14:43.864196 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:43.860484 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-bf51d6\"" Nov 02 23:14:43.901643 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:43.901602 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bf51d6-8f6bfb495-bqqqx"] Nov 02 23:14:43.907330 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:43.907299 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bf51d6-8f6bfb495-bqqqx" Nov 02 23:14:43.913319 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:43.913262 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m552\" (UniqueName: \"kubernetes.io/projected/d0d6dc88-070f-4891-8e58-8adb3606cde0-kube-api-access-7m552\") pod \"payload-pre-processing-e2e-api-a-bf51d6-6db79f996c-6c4dz\" (UID: \"d0d6dc88-070f-4891-8e58-8adb3606cde0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf51d6-6db79f996c-6c4dz" Nov 02 23:14:43.913502 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:43.913435 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0d6dc88-070f-4891-8e58-8adb3606cde0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bf51d6-6db79f996c-6c4dz\" (UID: \"d0d6dc88-070f-4891-8e58-8adb3606cde0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf51d6-6db79f996c-6c4dz" Nov 02 23:14:43.914341 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:43.914305 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0d6dc88-070f-4891-8e58-8adb3606cde0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bf51d6-6db79f996c-6c4dz\" (UID: \"d0d6dc88-070f-4891-8e58-8adb3606cde0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf51d6-6db79f996c-6c4dz" Nov 02 23:14:43.964154 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:43.964093 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bf51d6-8f6bfb495-bqqqx"] Nov 02 23:14:43.997385 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:43.997348 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m552\" (UniqueName: \"kubernetes.io/projected/d0d6dc88-070f-4891-8e58-8adb3606cde0-kube-api-access-7m552\") pod \"payload-pre-processing-e2e-api-a-bf51d6-6db79f996c-6c4dz\" (UID: \"d0d6dc88-070f-4891-8e58-8adb3606cde0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf51d6-6db79f996c-6c4dz" Nov 02 23:14:44.016733 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:44.015435 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cztk4\" (UniqueName: \"kubernetes.io/projected/6f7317cd-3f9f-4e50-a016-d18de7b5e8a6-kube-api-access-cztk4\") pod \"payload-processing-e2e-api-a-bf51d6-8f6bfb495-bqqqx\" (UID: \"6f7317cd-3f9f-4e50-a016-d18de7b5e8a6\") " pod="openshift-ingress/payload-processing-e2e-api-a-bf51d6-8f6bfb495-bqqqx" Nov 02 23:14:44.016733 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:44.015559 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7317cd-3f9f-4e50-a016-d18de7b5e8a6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bf51d6-8f6bfb495-bqqqx\" (UID: \"6f7317cd-3f9f-4e50-a016-d18de7b5e8a6\") " pod="openshift-ingress/payload-processing-e2e-api-a-bf51d6-8f6bfb495-bqqqx" Nov 02 23:14:44.075745 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:44.075715 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf51d6-6db79f996c-6c4dz" Nov 02 23:14:44.116958 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:44.116916 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cztk4\" (UniqueName: \"kubernetes.io/projected/6f7317cd-3f9f-4e50-a016-d18de7b5e8a6-kube-api-access-cztk4\") pod \"payload-processing-e2e-api-a-bf51d6-8f6bfb495-bqqqx\" (UID: \"6f7317cd-3f9f-4e50-a016-d18de7b5e8a6\") " pod="openshift-ingress/payload-processing-e2e-api-a-bf51d6-8f6bfb495-bqqqx" Nov 02 23:14:44.117171 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:44.116996 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7317cd-3f9f-4e50-a016-d18de7b5e8a6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bf51d6-8f6bfb495-bqqqx\" (UID: \"6f7317cd-3f9f-4e50-a016-d18de7b5e8a6\") " pod="openshift-ingress/payload-processing-e2e-api-a-bf51d6-8f6bfb495-bqqqx" Nov 02 23:14:44.117760 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:44.117730 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7317cd-3f9f-4e50-a016-d18de7b5e8a6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bf51d6-8f6bfb495-bqqqx\" (UID: \"6f7317cd-3f9f-4e50-a016-d18de7b5e8a6\") " pod="openshift-ingress/payload-processing-e2e-api-a-bf51d6-8f6bfb495-bqqqx" Nov 02 23:14:44.182299 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:44.182263 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cztk4\" (UniqueName: \"kubernetes.io/projected/6f7317cd-3f9f-4e50-a016-d18de7b5e8a6-kube-api-access-cztk4\") pod \"payload-processing-e2e-api-a-bf51d6-8f6bfb495-bqqqx\" (UID: \"6f7317cd-3f9f-4e50-a016-d18de7b5e8a6\") " pod="openshift-ingress/payload-processing-e2e-api-a-bf51d6-8f6bfb495-bqqqx" Nov 02 23:14:44.225609 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:44.225576 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bf51d6-8f6bfb495-bqqqx" Nov 02 23:14:44.482045 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:14:44.482011 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d6dc88_070f_4891_8e58_8adb3606cde0.slice/crio-bd88fb05238abb82553c9b351979a7709a3f620276307551413e80c77b2aa9d9 WatchSource:0}: Error finding container bd88fb05238abb82553c9b351979a7709a3f620276307551413e80c77b2aa9d9: Status 404 returned error can't find the container with id bd88fb05238abb82553c9b351979a7709a3f620276307551413e80c77b2aa9d9 Nov 02 23:14:44.483486 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:44.483464 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bf51d6-6db79f996c-6c4dz"] Nov 02 23:14:44.625007 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:14:44.624978 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f7317cd_3f9f_4e50_a016_d18de7b5e8a6.slice/crio-c218e6ef9f720ccacd50d392fa781444e4e52e2dfd01a1bd9b821e6d5d8b0b7e WatchSource:0}: Error finding container c218e6ef9f720ccacd50d392fa781444e4e52e2dfd01a1bd9b821e6d5d8b0b7e: Status 404 returned error can't find the container with id c218e6ef9f720ccacd50d392fa781444e4e52e2dfd01a1bd9b821e6d5d8b0b7e Nov 02 23:14:44.625892 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:44.625866 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bf51d6-8f6bfb495-bqqqx"] Nov 02 23:14:44.740487 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:44.740453 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bf51d6-8f6bfb495-bqqqx" event={"ID":"6f7317cd-3f9f-4e50-a016-d18de7b5e8a6","Type":"ContainerStarted","Data":"0468ea4cc006b47d022cb161a3f61aa38e7383dcea8e3b95f5d909809032662e"} Nov 02 23:14:44.740632 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:44.740515 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-bf51d6-8f6bfb495-bqqqx" Nov 02 23:14:44.740632 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:44.740540 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bf51d6-8f6bfb495-bqqqx" event={"ID":"6f7317cd-3f9f-4e50-a016-d18de7b5e8a6","Type":"ContainerStarted","Data":"c218e6ef9f720ccacd50d392fa781444e4e52e2dfd01a1bd9b821e6d5d8b0b7e"} Nov 02 23:14:44.742287 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:44.742257 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf51d6-6db79f996c-6c4dz" event={"ID":"d0d6dc88-070f-4891-8e58-8adb3606cde0","Type":"ContainerStarted","Data":"8277f05dbbcf33945e862601f2786d15c7f9922dc52f06cc6be202f715f241b2"} Nov 02 23:14:44.742409 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:44.742307 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf51d6-6db79f996c-6c4dz" event={"ID":"d0d6dc88-070f-4891-8e58-8adb3606cde0","Type":"ContainerStarted","Data":"bd88fb05238abb82553c9b351979a7709a3f620276307551413e80c77b2aa9d9"} Nov 02 23:14:44.742409 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:44.742360 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf51d6-6db79f996c-6c4dz" Nov 02 23:14:44.859241 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:44.859180 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-bf51d6-8f6bfb495-bqqqx" podStartSLOduration=1.859162543 podStartE2EDuration="1.859162543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:14:44.855307698 +0000 UTC m=+1571.093805423" watchObservedRunningTime="2026-11-02 23:14:44.859162543 +0000 UTC m=+1571.097660205" Nov 02 23:14:44.973457 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:44.973390 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf51d6-6db79f996c-6c4dz" podStartSLOduration=1.9733708490000001 podStartE2EDuration="1.973370849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:14:44.970940266 +0000 UTC m=+1571.209437929" watchObservedRunningTime="2026-11-02 23:14:44.973370849 +0000 UTC m=+1571.211868512" Nov 02 23:14:50.408097 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:50.408057 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-gfc2f" podUID="fd4d615f-189f-44b7-87cc-a7d64e9dfd81" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 23:14:55.343447 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:55.343421 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-gfc2f" Nov 02 23:14:55.469751 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:55.469711 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd4d615f-189f-44b7-87cc-a7d64e9dfd81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd4d615f-189f-44b7-87cc-a7d64e9dfd81-plugins-config-volume\") pod \"fd4d615f-189f-44b7-87cc-a7d64e9dfd81\" (UID: \"fd4d615f-189f-44b7-87cc-a7d64e9dfd81\") " Nov 02 23:14:55.469936 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:55.469777 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd4d615f-189f-44b7-87cc-a7d64e9dfd81-kube-api-access-5twlt\" (UniqueName: \"kubernetes.io/projected/fd4d615f-189f-44b7-87cc-a7d64e9dfd81-kube-api-access-5twlt\") pod \"fd4d615f-189f-44b7-87cc-a7d64e9dfd81\" (UID: \"fd4d615f-189f-44b7-87cc-a7d64e9dfd81\") " Nov 02 23:14:55.470747 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:55.470712 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd4d615f-189f-44b7-87cc-a7d64e9dfd81-plugins-config-volume" pod "fd4d615f-189f-44b7-87cc-a7d64e9dfd81" (UID: "fd4d615f-189f-44b7-87cc-a7d64e9dfd81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:14:55.473013 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:55.472978 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd4d615f-189f-44b7-87cc-a7d64e9dfd81-kube-api-access-5twlt" pod "fd4d615f-189f-44b7-87cc-a7d64e9dfd81" (UID: "fd4d615f-189f-44b7-87cc-a7d64e9dfd81"). InnerVolumeSpecName "kube-api-access-5twlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:14:55.571275 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:55.571239 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd4d615f-189f-44b7-87cc-a7d64e9dfd81-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:14:55.571275 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:55.571270 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5twlt\" (UniqueName: \"kubernetes.io/projected/fd4d615f-189f-44b7-87cc-a7d64e9dfd81-kube-api-access-5twlt\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:14:55.752502 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:55.752433 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf51d6-6db79f996c-6c4dz" Nov 02 23:14:55.752502 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:55.752500 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-bf51d6-8f6bfb495-bqqqx" Nov 02 23:14:55.835472 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:55.835431 5727 generic.go:356] "Generic (PLEG): container finished" podID="fd4d615f-189f-44b7-87cc-a7d64e9dfd81" containerID="85278bcda8cc8c37833df664ce38fa215093bd1fdfeb911c22148e64beb7bbb9" exitCode=0 Nov 02 23:14:55.835649 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:55.835493 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-gfc2f" Nov 02 23:14:55.835649 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:55.835519 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-gfc2f" event={"ID":"fd4d615f-189f-44b7-87cc-a7d64e9dfd81","Type":"ContainerDied","Data":"85278bcda8cc8c37833df664ce38fa215093bd1fdfeb911c22148e64beb7bbb9"} Nov 02 23:14:55.835649 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:55.835574 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-gfc2f" event={"ID":"fd4d615f-189f-44b7-87cc-a7d64e9dfd81","Type":"ContainerDied","Data":"4331efecc38136c526249bb28fd73872f77aee730d5674aebe09a07591d5ebab"} Nov 02 23:14:55.835649 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:55.835599 5727 scope.go:122] "RemoveContainer" containerID="85278bcda8cc8c37833df664ce38fa215093bd1fdfeb911c22148e64beb7bbb9" Nov 02 23:14:55.846956 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:55.846939 5727 scope.go:122] "RemoveContainer" containerID="85278bcda8cc8c37833df664ce38fa215093bd1fdfeb911c22148e64beb7bbb9" Nov 02 23:14:55.847260 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:14:55.847241 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85278bcda8cc8c37833df664ce38fa215093bd1fdfeb911c22148e64beb7bbb9\": container with ID starting with 85278bcda8cc8c37833df664ce38fa215093bd1fdfeb911c22148e64beb7bbb9 not found: ID does not exist" containerID="85278bcda8cc8c37833df664ce38fa215093bd1fdfeb911c22148e64beb7bbb9" Nov 02 23:14:55.847308 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:55.847269 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85278bcda8cc8c37833df664ce38fa215093bd1fdfeb911c22148e64beb7bbb9"} err="failed to get container status \"85278bcda8cc8c37833df664ce38fa215093bd1fdfeb911c22148e64beb7bbb9\": rpc error: code = NotFound desc = could not find container \"85278bcda8cc8c37833df664ce38fa215093bd1fdfeb911c22148e64beb7bbb9\": container with ID starting with 85278bcda8cc8c37833df664ce38fa215093bd1fdfeb911c22148e64beb7bbb9 not found: ID does not exist" Nov 02 23:14:56.100428 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:56.100398 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-gfc2f"] Nov 02 23:14:56.105201 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:56.105175 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-gfc2f"] Nov 02 23:14:57.846290 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:14:57.846254 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:14:57.884251 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:14:57.884220 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd4d615f-189f-44b7-87cc-a7d64e9dfd81" path="/var/lib/kubelet/pods/fd4d615f-189f-44b7-87cc-a7d64e9dfd81/volumes" Nov 02 23:15:23.613940 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:23.613909 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv"] Nov 02 23:15:23.614670 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:23.614655 5727 cpu_manager.go:414] "containerMap: removing container" podUID="fd4d615f-189f-44b7-87cc-a7d64e9dfd81" containerName="payload-pre-processing" Nov 02 23:15:23.614715 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:23.614672 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd4d615f-189f-44b7-87cc-a7d64e9dfd81" containerName="payload-pre-processing" Nov 02 23:15:23.614800 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:23.614791 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd4d615f-189f-44b7-87cc-a7d64e9dfd81" containerName="payload-pre-processing" Nov 02 23:15:23.618534 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:23.618513 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv" Nov 02 23:15:23.657924 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:23.657889 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d-maas-api-tls\") pod \"maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv\" (UID: \"11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv" Nov 02 23:15:23.658093 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:23.657933 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d-openshift-service-ca\") pod \"maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv\" (UID: \"11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv" Nov 02 23:15:23.658163 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:23.658098 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jk77\" (UniqueName: \"kubernetes.io/projected/11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d-kube-api-access-8jk77\") pod \"maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv\" (UID: \"11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv" Nov 02 23:15:23.658209 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:23.658186 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d-metrics-certs\") pod \"maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv\" (UID: \"11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv" Nov 02 23:15:23.670607 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:23.670584 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-62ab55-serving-cert\"" Nov 02 23:15:23.724344 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:23.724314 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv"] Nov 02 23:15:23.759859 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:23.759822 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d-maas-api-tls\") pod \"maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv\" (UID: \"11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv" Nov 02 23:15:23.760122 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:23.759878 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d-openshift-service-ca\") pod \"maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv\" (UID: \"11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv" Nov 02 23:15:23.760122 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:15:23.760001 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-serving-cert: secret "maas-api-e2e-api-b-62ab55-serving-cert" not found Nov 02 23:15:23.760122 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:23.760017 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jk77\" (UniqueName: \"kubernetes.io/projected/11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d-kube-api-access-8jk77\") pod \"maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv\" (UID: \"11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv" Nov 02 23:15:23.760122 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:15:23.760087 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d-maas-api-tls podName:11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d nodeName:}" failed. No retries permitted until 2026-11-02 23:15:24.26006528 +0000 UTC m=+1610.498562941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d-maas-api-tls") pod "maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv" (UID: "11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d") : secret "maas-api-e2e-api-b-62ab55-serving-cert" not found Nov 02 23:15:23.760547 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:23.760160 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d-metrics-certs\") pod \"maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv\" (UID: \"11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv" Nov 02 23:15:23.761286 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:23.760722 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d-openshift-service-ca\") pod \"maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv\" (UID: \"11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv" Nov 02 23:15:23.763490 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:23.763381 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d-metrics-certs\") pod \"maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv\" (UID: \"11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv" Nov 02 23:15:23.820944 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:23.820910 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jk77\" (UniqueName: \"kubernetes.io/projected/11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d-kube-api-access-8jk77\") pod \"maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv\" (UID: \"11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv" Nov 02 23:15:24.268793 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:24.268759 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d-maas-api-tls\") pod \"maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv\" (UID: \"11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv" Nov 02 23:15:24.271513 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:24.271480 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d-maas-api-tls\") pod \"maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv\" (UID: \"11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv" Nov 02 23:15:24.533494 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:24.533417 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv" Nov 02 23:15:24.924613 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:24.924578 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv"] Nov 02 23:15:24.925553 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:15:24.925522 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11fb6a0c_cf3a_4dbe_b1b4_a727c5b4c81d.slice/crio-e96d602d0d66749b9010267f1484f4aeaf0979f8e6cfbb0b34a27d1b4327e694 WatchSource:0}: Error finding container e96d602d0d66749b9010267f1484f4aeaf0979f8e6cfbb0b34a27d1b4327e694: Status 404 returned error can't find the container with id e96d602d0d66749b9010267f1484f4aeaf0979f8e6cfbb0b34a27d1b4327e694 Nov 02 23:15:25.083171 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:25.076649 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv" event={"ID":"11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d","Type":"ContainerStarted","Data":"e96d602d0d66749b9010267f1484f4aeaf0979f8e6cfbb0b34a27d1b4327e694"} Nov 02 23:15:26.096250 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:26.096211 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv" event={"ID":"11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d","Type":"ContainerStarted","Data":"ea17c9f09251e27128588ac820409f69752d64cc29e8fd396c4c9c3cd521d3b4"} Nov 02 23:15:26.096862 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:26.096836 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv" Nov 02 23:15:26.215892 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:26.215833 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv" podStartSLOduration=2.420493278 podStartE2EDuration="3.215820221s" totalImagesPullingTime="795.326943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:15:26.214089531 +0000 UTC m=+1612.452587213" watchObservedRunningTime="2026-11-02 23:15:26.215820221 +0000 UTC m=+1612.454317925" Nov 02 23:15:27.846892 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:15:27.846851 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:15:29.240698 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:29.240667 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bf51d6-666868699c-h548g"] Nov 02 23:15:29.245356 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:29.245333 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf51d6-666868699c-h548g" Nov 02 23:15:29.304295 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:29.304265 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bf51d6-666868699c-h548g"] Nov 02 23:15:29.348011 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:29.347954 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-bf51d6-serving-cert\"" Nov 02 23:15:29.355280 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:29.355250 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96e0cb73-16c1-4cb9-9e37-8276276f4b44-openshift-service-ca\") pod \"maas-api-e2e-api-a-bf51d6-666868699c-h548g\" (UID: \"96e0cb73-16c1-4cb9-9e37-8276276f4b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf51d6-666868699c-h548g" Nov 02 23:15:29.355526 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:29.355506 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96e0cb73-16c1-4cb9-9e37-8276276f4b44-metrics-certs\") pod \"maas-api-e2e-api-a-bf51d6-666868699c-h548g\" (UID: \"96e0cb73-16c1-4cb9-9e37-8276276f4b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf51d6-666868699c-h548g" Nov 02 23:15:29.355587 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:29.355558 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4hw2\" (UniqueName: \"kubernetes.io/projected/96e0cb73-16c1-4cb9-9e37-8276276f4b44-kube-api-access-c4hw2\") pod \"maas-api-e2e-api-a-bf51d6-666868699c-h548g\" (UID: \"96e0cb73-16c1-4cb9-9e37-8276276f4b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf51d6-666868699c-h548g" Nov 02 23:15:29.355646 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:29.355582 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96e0cb73-16c1-4cb9-9e37-8276276f4b44-maas-api-tls\") pod \"maas-api-e2e-api-a-bf51d6-666868699c-h548g\" (UID: \"96e0cb73-16c1-4cb9-9e37-8276276f4b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf51d6-666868699c-h548g" Nov 02 23:15:29.456836 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:29.456806 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96e0cb73-16c1-4cb9-9e37-8276276f4b44-openshift-service-ca\") pod \"maas-api-e2e-api-a-bf51d6-666868699c-h548g\" (UID: \"96e0cb73-16c1-4cb9-9e37-8276276f4b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf51d6-666868699c-h548g" Nov 02 23:15:29.457030 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:29.456986 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96e0cb73-16c1-4cb9-9e37-8276276f4b44-metrics-certs\") pod \"maas-api-e2e-api-a-bf51d6-666868699c-h548g\" (UID: \"96e0cb73-16c1-4cb9-9e37-8276276f4b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf51d6-666868699c-h548g" Nov 02 23:15:29.457030 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:29.457027 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4hw2\" (UniqueName: \"kubernetes.io/projected/96e0cb73-16c1-4cb9-9e37-8276276f4b44-kube-api-access-c4hw2\") pod \"maas-api-e2e-api-a-bf51d6-666868699c-h548g\" (UID: \"96e0cb73-16c1-4cb9-9e37-8276276f4b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf51d6-666868699c-h548g" Nov 02 23:15:29.457577 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:29.457046 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96e0cb73-16c1-4cb9-9e37-8276276f4b44-maas-api-tls\") pod \"maas-api-e2e-api-a-bf51d6-666868699c-h548g\" (UID: \"96e0cb73-16c1-4cb9-9e37-8276276f4b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf51d6-666868699c-h548g" Nov 02 23:15:29.457577 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:15:29.457195 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-bf51d6-serving-cert: secret "maas-api-e2e-api-a-bf51d6-serving-cert" not found Nov 02 23:15:29.457577 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:15:29.457259 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96e0cb73-16c1-4cb9-9e37-8276276f4b44-maas-api-tls podName:96e0cb73-16c1-4cb9-9e37-8276276f4b44 nodeName:}" failed. No retries permitted until 2026-11-02 23:15:29.9572442 +0000 UTC m=+1616.195741841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/96e0cb73-16c1-4cb9-9e37-8276276f4b44-maas-api-tls") pod "maas-api-e2e-api-a-bf51d6-666868699c-h548g" (UID: "96e0cb73-16c1-4cb9-9e37-8276276f4b44") : secret "maas-api-e2e-api-a-bf51d6-serving-cert" not found Nov 02 23:15:29.457780 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:29.457652 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96e0cb73-16c1-4cb9-9e37-8276276f4b44-openshift-service-ca\") pod \"maas-api-e2e-api-a-bf51d6-666868699c-h548g\" (UID: \"96e0cb73-16c1-4cb9-9e37-8276276f4b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf51d6-666868699c-h548g" Nov 02 23:15:29.459653 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:29.459628 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96e0cb73-16c1-4cb9-9e37-8276276f4b44-metrics-certs\") pod \"maas-api-e2e-api-a-bf51d6-666868699c-h548g\" (UID: \"96e0cb73-16c1-4cb9-9e37-8276276f4b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf51d6-666868699c-h548g" Nov 02 23:15:29.518357 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:29.518280 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4hw2\" (UniqueName: \"kubernetes.io/projected/96e0cb73-16c1-4cb9-9e37-8276276f4b44-kube-api-access-c4hw2\") pod \"maas-api-e2e-api-a-bf51d6-666868699c-h548g\" (UID: \"96e0cb73-16c1-4cb9-9e37-8276276f4b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf51d6-666868699c-h548g" Nov 02 23:15:29.761103 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:29.761066 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8e34df-b48f777f9-t7tg2"] Nov 02 23:15:29.971404 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:29.970017 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96e0cb73-16c1-4cb9-9e37-8276276f4b44-maas-api-tls\") pod \"maas-api-e2e-api-a-bf51d6-666868699c-h548g\" (UID: \"96e0cb73-16c1-4cb9-9e37-8276276f4b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf51d6-666868699c-h548g" Nov 02 23:15:30.002175 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:30.001055 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96e0cb73-16c1-4cb9-9e37-8276276f4b44-maas-api-tls\") pod \"maas-api-e2e-api-a-bf51d6-666868699c-h548g\" (UID: \"96e0cb73-16c1-4cb9-9e37-8276276f4b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf51d6-666868699c-h548g" Nov 02 23:15:30.161533 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:30.161504 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf51d6-666868699c-h548g" Nov 02 23:15:30.559047 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:30.559016 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bf51d6-666868699c-h548g"] Nov 02 23:15:30.559513 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:15:30.559227 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e0cb73_16c1_4cb9_9e37_8276276f4b44.slice/crio-6a34f4c2139e5f5a5c5aa89f301be8763c753850c5ba5e8afccfaf84735ef962 WatchSource:0}: Error finding container 6a34f4c2139e5f5a5c5aa89f301be8763c753850c5ba5e8afccfaf84735ef962: Status 404 returned error can't find the container with id 6a34f4c2139e5f5a5c5aa89f301be8763c753850c5ba5e8afccfaf84735ef962 Nov 02 23:15:31.155211 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:31.155173 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf51d6-666868699c-h548g" event={"ID":"96e0cb73-16c1-4cb9-9e37-8276276f4b44","Type":"ContainerStarted","Data":"6a34f4c2139e5f5a5c5aa89f301be8763c753850c5ba5e8afccfaf84735ef962"} Nov 02 23:15:32.109713 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:32.109686 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv" Nov 02 23:15:32.166807 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:32.166757 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf51d6-666868699c-h548g" event={"ID":"96e0cb73-16c1-4cb9-9e37-8276276f4b44","Type":"ContainerStarted","Data":"46ecd45e2bbda2c52ae4fe8caacf70487116714e73d1f9b792604725fb46efc2"} Nov 02 23:15:32.166981 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:32.166834 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf51d6-666868699c-h548g" Nov 02 23:15:32.346506 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:32.346422 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf51d6-666868699c-h548g" podStartSLOduration=2.542693221 podStartE2EDuration="3.346403923s" totalImagesPullingTime="803.710702ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:15:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:15:32.342147377 +0000 UTC m=+1618.580645062" watchObservedRunningTime="2026-11-02 23:15:32.346403923 +0000 UTC m=+1618.584901585" Nov 02 23:15:34.762303 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:34.762259 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8e34df-b48f777f9-t7tg2" podUID="875c0021-b1a3-45b3-81a4-da889bf77c04" containerName="payload-pre-processing" containerID="cri-o://19320e3a37ab578eea617d7d421768b17a3998f35567961db87f9dfe536894ec" gracePeriod=25 Nov 02 23:15:35.068970 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:35.068944 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8e34df-b48f777f9-t7tg2" Nov 02 23:15:35.217899 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:35.217861 5727 generic.go:356] "Generic (PLEG): container finished" podID="875c0021-b1a3-45b3-81a4-da889bf77c04" containerID="19320e3a37ab578eea617d7d421768b17a3998f35567961db87f9dfe536894ec" exitCode=0 Nov 02 23:15:35.218080 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:35.217951 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8e34df-b48f777f9-t7tg2" event={"ID":"875c0021-b1a3-45b3-81a4-da889bf77c04","Type":"ContainerDied","Data":"19320e3a37ab578eea617d7d421768b17a3998f35567961db87f9dfe536894ec"} Nov 02 23:15:35.218080 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:35.218009 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8e34df-b48f777f9-t7tg2" Nov 02 23:15:35.218080 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:35.218030 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8e34df-b48f777f9-t7tg2" event={"ID":"875c0021-b1a3-45b3-81a4-da889bf77c04","Type":"ContainerDied","Data":"8c9bfa10c649ee5c2d4df55fa4527d7c337e5aa959515e87245330c86abb1e63"} Nov 02 23:15:35.218080 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:35.218070 5727 scope.go:122] "RemoveContainer" containerID="19320e3a37ab578eea617d7d421768b17a3998f35567961db87f9dfe536894ec" Nov 02 23:15:35.234273 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:35.234238 5727 scope.go:122] "RemoveContainer" containerID="19320e3a37ab578eea617d7d421768b17a3998f35567961db87f9dfe536894ec" Nov 02 23:15:35.234593 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:15:35.234575 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19320e3a37ab578eea617d7d421768b17a3998f35567961db87f9dfe536894ec\": container with ID starting with 19320e3a37ab578eea617d7d421768b17a3998f35567961db87f9dfe536894ec not found: ID does not exist" containerID="19320e3a37ab578eea617d7d421768b17a3998f35567961db87f9dfe536894ec" Nov 02 23:15:35.234673 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:35.234603 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19320e3a37ab578eea617d7d421768b17a3998f35567961db87f9dfe536894ec"} err="failed to get container status \"19320e3a37ab578eea617d7d421768b17a3998f35567961db87f9dfe536894ec\": rpc error: code = NotFound desc = could not find container \"19320e3a37ab578eea617d7d421768b17a3998f35567961db87f9dfe536894ec\": container with ID starting with 19320e3a37ab578eea617d7d421768b17a3998f35567961db87f9dfe536894ec not found: ID does not exist" Nov 02 23:15:35.245354 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:35.245328 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/875c0021-b1a3-45b3-81a4-da889bf77c04-kube-api-access-7fjt6\" (UniqueName: \"kubernetes.io/projected/875c0021-b1a3-45b3-81a4-da889bf77c04-kube-api-access-7fjt6\") pod \"875c0021-b1a3-45b3-81a4-da889bf77c04\" (UID: \"875c0021-b1a3-45b3-81a4-da889bf77c04\") " Nov 02 23:15:35.245651 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:35.245631 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/875c0021-b1a3-45b3-81a4-da889bf77c04-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/875c0021-b1a3-45b3-81a4-da889bf77c04-plugins-config-volume\") pod \"875c0021-b1a3-45b3-81a4-da889bf77c04\" (UID: \"875c0021-b1a3-45b3-81a4-da889bf77c04\") " Nov 02 23:15:35.246072 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:35.245956 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/875c0021-b1a3-45b3-81a4-da889bf77c04-plugins-config-volume" pod "875c0021-b1a3-45b3-81a4-da889bf77c04" (UID: "875c0021-b1a3-45b3-81a4-da889bf77c04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:15:35.246215 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:35.246198 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/875c0021-b1a3-45b3-81a4-da889bf77c04-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:35.247560 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:35.247532 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/875c0021-b1a3-45b3-81a4-da889bf77c04-kube-api-access-7fjt6" pod "875c0021-b1a3-45b3-81a4-da889bf77c04" (UID: "875c0021-b1a3-45b3-81a4-da889bf77c04"). InnerVolumeSpecName "kube-api-access-7fjt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:15:35.347062 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:35.347033 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fjt6\" (UniqueName: \"kubernetes.io/projected/875c0021-b1a3-45b3-81a4-da889bf77c04-kube-api-access-7fjt6\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:35.643931 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:35.643833 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8e34df-b48f777f9-t7tg2"] Nov 02 23:15:35.647768 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:35.647742 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8e34df-b48f777f9-t7tg2"] Nov 02 23:15:35.885463 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:35.885423 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="875c0021-b1a3-45b3-81a4-da889bf77c04" path="/var/lib/kubelet/pods/875c0021-b1a3-45b3-81a4-da889bf77c04/volumes" Nov 02 23:15:38.184729 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:38.184699 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf51d6-666868699c-h548g" Nov 02 23:15:57.846535 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:15:57.846500 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:15:59.677806 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:59.677767 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k"] Nov 02 23:15:59.678365 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:59.678036 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k" podUID="af82c965-7e41-4e03-97fb-f84c11f0b73e" containerName="maas-api" containerID="cri-o://39c96e08c90b15dcf5e15ada95a6adca8c046500e0cc5aaceb1191f47eb24342" gracePeriod=30 Nov 02 23:15:59.984373 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:15:59.984343 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k" Nov 02 23:16:00.110600 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:00.110563 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af82c965-7e41-4e03-97fb-f84c11f0b73e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af82c965-7e41-4e03-97fb-f84c11f0b73e-openshift-service-ca\") pod \"af82c965-7e41-4e03-97fb-f84c11f0b73e\" (UID: \"af82c965-7e41-4e03-97fb-f84c11f0b73e\") " Nov 02 23:16:00.110798 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:00.110618 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af82c965-7e41-4e03-97fb-f84c11f0b73e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/af82c965-7e41-4e03-97fb-f84c11f0b73e-metrics-certs\") pod \"af82c965-7e41-4e03-97fb-f84c11f0b73e\" (UID: \"af82c965-7e41-4e03-97fb-f84c11f0b73e\") " Nov 02 23:16:00.110798 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:00.110651 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af82c965-7e41-4e03-97fb-f84c11f0b73e-kube-api-access-qzvlc\" (UniqueName: \"kubernetes.io/projected/af82c965-7e41-4e03-97fb-f84c11f0b73e-kube-api-access-qzvlc\") pod \"af82c965-7e41-4e03-97fb-f84c11f0b73e\" (UID: \"af82c965-7e41-4e03-97fb-f84c11f0b73e\") " Nov 02 23:16:00.110914 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:00.110807 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af82c965-7e41-4e03-97fb-f84c11f0b73e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af82c965-7e41-4e03-97fb-f84c11f0b73e-maas-api-tls\") pod \"af82c965-7e41-4e03-97fb-f84c11f0b73e\" (UID: \"af82c965-7e41-4e03-97fb-f84c11f0b73e\") " Nov 02 23:16:00.111708 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:00.111678 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af82c965-7e41-4e03-97fb-f84c11f0b73e-openshift-service-ca" pod "af82c965-7e41-4e03-97fb-f84c11f0b73e" (UID: "af82c965-7e41-4e03-97fb-f84c11f0b73e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:16:00.112265 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:00.112217 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af82c965-7e41-4e03-97fb-f84c11f0b73e-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:00.115332 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:00.115304 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af82c965-7e41-4e03-97fb-f84c11f0b73e-kube-api-access-qzvlc" pod "af82c965-7e41-4e03-97fb-f84c11f0b73e" (UID: "af82c965-7e41-4e03-97fb-f84c11f0b73e"). InnerVolumeSpecName "kube-api-access-qzvlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:16:00.116635 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:00.116608 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af82c965-7e41-4e03-97fb-f84c11f0b73e-maas-api-tls" pod "af82c965-7e41-4e03-97fb-f84c11f0b73e" (UID: "af82c965-7e41-4e03-97fb-f84c11f0b73e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:16:00.116730 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:00.116710 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af82c965-7e41-4e03-97fb-f84c11f0b73e-metrics-certs" pod "af82c965-7e41-4e03-97fb-f84c11f0b73e" (UID: "af82c965-7e41-4e03-97fb-f84c11f0b73e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:16:00.214119 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:00.214035 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af82c965-7e41-4e03-97fb-f84c11f0b73e-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:00.214119 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:00.214074 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af82c965-7e41-4e03-97fb-f84c11f0b73e-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:00.214119 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:00.214089 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzvlc\" (UniqueName: \"kubernetes.io/projected/af82c965-7e41-4e03-97fb-f84c11f0b73e-kube-api-access-qzvlc\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:00.434445 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:00.434408 5727 generic.go:356] "Generic (PLEG): container finished" podID="af82c965-7e41-4e03-97fb-f84c11f0b73e" containerID="39c96e08c90b15dcf5e15ada95a6adca8c046500e0cc5aaceb1191f47eb24342" exitCode=0 Nov 02 23:16:00.434609 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:00.434448 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k" Nov 02 23:16:00.434609 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:00.434491 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k" event={"ID":"af82c965-7e41-4e03-97fb-f84c11f0b73e","Type":"ContainerDied","Data":"39c96e08c90b15dcf5e15ada95a6adca8c046500e0cc5aaceb1191f47eb24342"} Nov 02 23:16:00.434609 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:00.434553 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k" event={"ID":"af82c965-7e41-4e03-97fb-f84c11f0b73e","Type":"ContainerDied","Data":"b2fbf071466aa15bae0f45b64c9f04bd7d26e5dfd2c317afffc237f6c7422192"} Nov 02 23:16:00.434609 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:00.434581 5727 scope.go:122] "RemoveContainer" containerID="39c96e08c90b15dcf5e15ada95a6adca8c046500e0cc5aaceb1191f47eb24342" Nov 02 23:16:00.446269 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:00.446250 5727 scope.go:122] "RemoveContainer" containerID="39c96e08c90b15dcf5e15ada95a6adca8c046500e0cc5aaceb1191f47eb24342" Nov 02 23:16:00.446534 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:16:00.446516 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39c96e08c90b15dcf5e15ada95a6adca8c046500e0cc5aaceb1191f47eb24342\": container with ID starting with 39c96e08c90b15dcf5e15ada95a6adca8c046500e0cc5aaceb1191f47eb24342 not found: ID does not exist" containerID="39c96e08c90b15dcf5e15ada95a6adca8c046500e0cc5aaceb1191f47eb24342" Nov 02 23:16:00.446607 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:00.446545 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39c96e08c90b15dcf5e15ada95a6adca8c046500e0cc5aaceb1191f47eb24342"} err="failed to get container status \"39c96e08c90b15dcf5e15ada95a6adca8c046500e0cc5aaceb1191f47eb24342\": rpc error: code = NotFound desc = could not find container \"39c96e08c90b15dcf5e15ada95a6adca8c046500e0cc5aaceb1191f47eb24342\": container with ID starting with 39c96e08c90b15dcf5e15ada95a6adca8c046500e0cc5aaceb1191f47eb24342 not found: ID does not exist" Nov 02 23:16:00.600805 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:00.600771 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k"] Nov 02 23:16:00.606017 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:00.605994 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8e34df-6445cbcd6c-bvk8k"] Nov 02 23:16:00.880031 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:00.879935 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj"] Nov 02 23:16:00.880988 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:00.880914 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj" podUID="9be2be64-8a5a-4275-a973-1a78b432c62f" containerName="maas-api" containerID="cri-o://456d6f1876852d656d9ca1e8367c83086ed2d39afafa0ddd5bdd6fbe5d942b41" gracePeriod=30 Nov 02 23:16:01.207277 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:01.207250 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj" Nov 02 23:16:01.340351 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:01.340312 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9be2be64-8a5a-4275-a973-1a78b432c62f-kube-api-access-whjnm\" (UniqueName: \"kubernetes.io/projected/9be2be64-8a5a-4275-a973-1a78b432c62f-kube-api-access-whjnm\") pod \"9be2be64-8a5a-4275-a973-1a78b432c62f\" (UID: \"9be2be64-8a5a-4275-a973-1a78b432c62f\") " Nov 02 23:16:01.340558 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:01.340432 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9be2be64-8a5a-4275-a973-1a78b432c62f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9be2be64-8a5a-4275-a973-1a78b432c62f-maas-api-tls\") pod \"9be2be64-8a5a-4275-a973-1a78b432c62f\" (UID: \"9be2be64-8a5a-4275-a973-1a78b432c62f\") " Nov 02 23:16:01.340558 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:01.340511 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9be2be64-8a5a-4275-a973-1a78b432c62f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9be2be64-8a5a-4275-a973-1a78b432c62f-openshift-service-ca\") pod \"9be2be64-8a5a-4275-a973-1a78b432c62f\" (UID: \"9be2be64-8a5a-4275-a973-1a78b432c62f\") " Nov 02 23:16:01.340682 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:01.340595 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9be2be64-8a5a-4275-a973-1a78b432c62f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9be2be64-8a5a-4275-a973-1a78b432c62f-metrics-certs\") pod \"9be2be64-8a5a-4275-a973-1a78b432c62f\" (UID: \"9be2be64-8a5a-4275-a973-1a78b432c62f\") " Nov 02 23:16:01.349540 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:01.349479 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9be2be64-8a5a-4275-a973-1a78b432c62f-openshift-service-ca" pod "9be2be64-8a5a-4275-a973-1a78b432c62f" (UID: "9be2be64-8a5a-4275-a973-1a78b432c62f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:16:01.365693 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:01.365617 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9be2be64-8a5a-4275-a973-1a78b432c62f-kube-api-access-whjnm" pod "9be2be64-8a5a-4275-a973-1a78b432c62f" (UID: "9be2be64-8a5a-4275-a973-1a78b432c62f"). InnerVolumeSpecName "kube-api-access-whjnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:16:01.365905 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:01.365710 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9be2be64-8a5a-4275-a973-1a78b432c62f-metrics-certs" pod "9be2be64-8a5a-4275-a973-1a78b432c62f" (UID: "9be2be64-8a5a-4275-a973-1a78b432c62f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:16:01.367873 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:01.367699 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9be2be64-8a5a-4275-a973-1a78b432c62f-maas-api-tls" pod "9be2be64-8a5a-4275-a973-1a78b432c62f" (UID: "9be2be64-8a5a-4275-a973-1a78b432c62f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:16:01.455960 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:01.450517 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whjnm\" (UniqueName: \"kubernetes.io/projected/9be2be64-8a5a-4275-a973-1a78b432c62f-kube-api-access-whjnm\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:01.455960 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:01.450552 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9be2be64-8a5a-4275-a973-1a78b432c62f-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:01.455960 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:01.450567 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9be2be64-8a5a-4275-a973-1a78b432c62f-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:01.455960 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:01.450578 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9be2be64-8a5a-4275-a973-1a78b432c62f-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:01.461534 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:01.461494 5727 generic.go:356] "Generic (PLEG): container finished" podID="9be2be64-8a5a-4275-a973-1a78b432c62f" containerID="456d6f1876852d656d9ca1e8367c83086ed2d39afafa0ddd5bdd6fbe5d942b41" exitCode=0 Nov 02 23:16:01.461763 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:01.461565 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj" Nov 02 23:16:01.461763 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:01.461638 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj" event={"ID":"9be2be64-8a5a-4275-a973-1a78b432c62f","Type":"ContainerDied","Data":"456d6f1876852d656d9ca1e8367c83086ed2d39afafa0ddd5bdd6fbe5d942b41"} Nov 02 23:16:01.461763 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:01.461685 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj" event={"ID":"9be2be64-8a5a-4275-a973-1a78b432c62f","Type":"ContainerDied","Data":"d74f24c7c085356db06f509bfce35dfcd1e14874b741e651af8c06ae0fe6905d"} Nov 02 23:16:01.461763 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:01.461715 5727 scope.go:122] "RemoveContainer" containerID="456d6f1876852d656d9ca1e8367c83086ed2d39afafa0ddd5bdd6fbe5d942b41" Nov 02 23:16:01.486336 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:01.486313 5727 scope.go:122] "RemoveContainer" containerID="456d6f1876852d656d9ca1e8367c83086ed2d39afafa0ddd5bdd6fbe5d942b41" Nov 02 23:16:01.486661 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:16:01.486639 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"456d6f1876852d656d9ca1e8367c83086ed2d39afafa0ddd5bdd6fbe5d942b41\": container with ID starting with 456d6f1876852d656d9ca1e8367c83086ed2d39afafa0ddd5bdd6fbe5d942b41 not found: ID does not exist" containerID="456d6f1876852d656d9ca1e8367c83086ed2d39afafa0ddd5bdd6fbe5d942b41" Nov 02 23:16:01.486727 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:01.486671 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"456d6f1876852d656d9ca1e8367c83086ed2d39afafa0ddd5bdd6fbe5d942b41"} err="failed to get container status \"456d6f1876852d656d9ca1e8367c83086ed2d39afafa0ddd5bdd6fbe5d942b41\": rpc error: code = NotFound desc = could not find container \"456d6f1876852d656d9ca1e8367c83086ed2d39afafa0ddd5bdd6fbe5d942b41\": container with ID starting with 456d6f1876852d656d9ca1e8367c83086ed2d39afafa0ddd5bdd6fbe5d942b41 not found: ID does not exist" Nov 02 23:16:01.629171 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:01.629139 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj"] Nov 02 23:16:01.632767 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:01.632744 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-qsvpj"] Nov 02 23:16:01.887980 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:01.887937 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9be2be64-8a5a-4275-a973-1a78b432c62f" path="/var/lib/kubelet/pods/9be2be64-8a5a-4275-a973-1a78b432c62f/volumes" Nov 02 23:16:01.888643 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:01.888621 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af82c965-7e41-4e03-97fb-f84c11f0b73e" path="/var/lib/kubelet/pods/af82c965-7e41-4e03-97fb-f84c11f0b73e/volumes" Nov 02 23:16:27.269390 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:27.269352 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6"] Nov 02 23:16:27.269811 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:27.269660 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" podUID="57d13cfb-07f6-4709-8ceb-068e0528e721" containerName="istio-proxy" containerID="cri-o://c0adaf38da9467dc2dac487c70aebe1217051f5e4728f67d32efe2602dce3eb4" gracePeriod=30 Nov 02 23:16:27.847822 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:16:27.847720 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:16:32.446476 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.446436 5727 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 23:16:32.447040 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.446498 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" podUID="57d13cfb-07f6-4709-8ceb-068e0528e721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 23:16:32.595482 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.595457 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:16:32.714007 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.713970 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57d13cfb-07f6-4709-8ceb-068e0528e721-istio-token\" (UniqueName: \"kubernetes.io/projected/57d13cfb-07f6-4709-8ceb-068e0528e721-istio-token\") pod \"57d13cfb-07f6-4709-8ceb-068e0528e721\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " Nov 02 23:16:32.714007 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.714010 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57d13cfb-07f6-4709-8ceb-068e0528e721-kube-api-access-ds99n\" (UniqueName: \"kubernetes.io/projected/57d13cfb-07f6-4709-8ceb-068e0528e721-kube-api-access-ds99n\") pod \"57d13cfb-07f6-4709-8ceb-068e0528e721\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " Nov 02 23:16:32.714287 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.714036 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-credential-socket\") pod \"57d13cfb-07f6-4709-8ceb-068e0528e721\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " Nov 02 23:16:32.714287 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.714063 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-istio-envoy\") pod \"57d13cfb-07f6-4709-8ceb-068e0528e721\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " Nov 02 23:16:32.714287 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.714090 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57d13cfb-07f6-4709-8ceb-068e0528e721-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57d13cfb-07f6-4709-8ceb-068e0528e721-istio-podinfo\") pod \"57d13cfb-07f6-4709-8ceb-068e0528e721\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " Nov 02 23:16:32.714287 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.714183 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-istio-data\") pod \"57d13cfb-07f6-4709-8ceb-068e0528e721\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " Nov 02 23:16:32.714287 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.714233 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57d13cfb-07f6-4709-8ceb-068e0528e721-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57d13cfb-07f6-4709-8ceb-068e0528e721-istiod-ca-cert\") pod \"57d13cfb-07f6-4709-8ceb-068e0528e721\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " Nov 02 23:16:32.714623 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.714405 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-workload-socket\") pod \"57d13cfb-07f6-4709-8ceb-068e0528e721\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " Nov 02 23:16:32.714623 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.714427 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-credential-socket" pod "57d13cfb-07f6-4709-8ceb-068e0528e721" (UID: "57d13cfb-07f6-4709-8ceb-068e0528e721"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:32.714623 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.714467 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-workload-certs\") pod \"57d13cfb-07f6-4709-8ceb-068e0528e721\" (UID: \"57d13cfb-07f6-4709-8ceb-068e0528e721\") " Nov 02 23:16:32.714782 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.714749 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-istio-data" pod "57d13cfb-07f6-4709-8ceb-068e0528e721" (UID: "57d13cfb-07f6-4709-8ceb-068e0528e721"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:32.714782 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.714764 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-workload-socket" pod "57d13cfb-07f6-4709-8ceb-068e0528e721" (UID: "57d13cfb-07f6-4709-8ceb-068e0528e721"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:32.715068 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.714926 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-workload-certs" pod "57d13cfb-07f6-4709-8ceb-068e0528e721" (UID: "57d13cfb-07f6-4709-8ceb-068e0528e721"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:32.715256 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.715228 5727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-workload-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:32.715256 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.715255 5727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-workload-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:32.715405 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.715271 5727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-credential-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:32.715405 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.715288 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57d13cfb-07f6-4709-8ceb-068e0528e721-istiod-ca-cert" pod "57d13cfb-07f6-4709-8ceb-068e0528e721" (UID: "57d13cfb-07f6-4709-8ceb-068e0528e721"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:16:32.715405 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.715330 5727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-istio-data\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:32.717103 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.716899 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-istio-envoy" pod "57d13cfb-07f6-4709-8ceb-068e0528e721" (UID: "57d13cfb-07f6-4709-8ceb-068e0528e721"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:32.717103 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.717017 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57d13cfb-07f6-4709-8ceb-068e0528e721-istio-podinfo" pod "57d13cfb-07f6-4709-8ceb-068e0528e721" (UID: "57d13cfb-07f6-4709-8ceb-068e0528e721"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:16:32.717103 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.717049 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57d13cfb-07f6-4709-8ceb-068e0528e721-kube-api-access-ds99n" pod "57d13cfb-07f6-4709-8ceb-068e0528e721" (UID: "57d13cfb-07f6-4709-8ceb-068e0528e721"). InnerVolumeSpecName "kube-api-access-ds99n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:16:32.717368 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.717266 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57d13cfb-07f6-4709-8ceb-068e0528e721-istio-token" pod "57d13cfb-07f6-4709-8ceb-068e0528e721" (UID: "57d13cfb-07f6-4709-8ceb-068e0528e721"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:16:32.903811 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.903704 5727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57d13cfb-07f6-4709-8ceb-068e0528e721-istio-token\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:32.903811 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.903745 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds99n\" (UniqueName: \"kubernetes.io/projected/57d13cfb-07f6-4709-8ceb-068e0528e721-kube-api-access-ds99n\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:32.903811 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.903761 5727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57d13cfb-07f6-4709-8ceb-068e0528e721-istio-envoy\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:32.903811 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.903775 5727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57d13cfb-07f6-4709-8ceb-068e0528e721-istio-podinfo\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:32.903811 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.903791 5727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57d13cfb-07f6-4709-8ceb-068e0528e721-istiod-ca-cert\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:32.913298 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.911852 5727 generic.go:356] "Generic (PLEG): container finished" podID="57d13cfb-07f6-4709-8ceb-068e0528e721" containerID="c0adaf38da9467dc2dac487c70aebe1217051f5e4728f67d32efe2602dce3eb4" exitCode=0 Nov 02 23:16:32.913298 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.912096 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" event={"ID":"57d13cfb-07f6-4709-8ceb-068e0528e721","Type":"ContainerDied","Data":"c0adaf38da9467dc2dac487c70aebe1217051f5e4728f67d32efe2602dce3eb4"} Nov 02 23:16:32.913298 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.912193 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" event={"ID":"57d13cfb-07f6-4709-8ceb-068e0528e721","Type":"ContainerDied","Data":"68cb10bf0afd4ae1de3cc0430fc1d5284a6f6ab7a9159db6f37013c19bc356a3"} Nov 02 23:16:32.913298 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.912225 5727 scope.go:122] "RemoveContainer" containerID="c0adaf38da9467dc2dac487c70aebe1217051f5e4728f67d32efe2602dce3eb4" Nov 02 23:16:32.913298 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.912395 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6" Nov 02 23:16:32.934782 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.934428 5727 scope.go:122] "RemoveContainer" containerID="c0adaf38da9467dc2dac487c70aebe1217051f5e4728f67d32efe2602dce3eb4" Nov 02 23:16:32.935055 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:16:32.935021 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0adaf38da9467dc2dac487c70aebe1217051f5e4728f67d32efe2602dce3eb4\": container with ID starting with c0adaf38da9467dc2dac487c70aebe1217051f5e4728f67d32efe2602dce3eb4 not found: ID does not exist" containerID="c0adaf38da9467dc2dac487c70aebe1217051f5e4728f67d32efe2602dce3eb4" Nov 02 23:16:32.935213 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:32.935077 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0adaf38da9467dc2dac487c70aebe1217051f5e4728f67d32efe2602dce3eb4"} err="failed to get container status \"c0adaf38da9467dc2dac487c70aebe1217051f5e4728f67d32efe2602dce3eb4\": rpc error: code = NotFound desc = could not find container \"c0adaf38da9467dc2dac487c70aebe1217051f5e4728f67d32efe2602dce3eb4\": container with ID starting with c0adaf38da9467dc2dac487c70aebe1217051f5e4728f67d32efe2602dce3eb4 not found: ID does not exist" Nov 02 23:16:33.085924 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:33.085858 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6"] Nov 02 23:16:33.090056 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:33.089985 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-vjbv6"] Nov 02 23:16:33.885288 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:33.885256 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57d13cfb-07f6-4709-8ceb-068e0528e721" path="/var/lib/kubelet/pods/57d13cfb-07f6-4709-8ceb-068e0528e721/volumes" Nov 02 23:16:36.031413 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:36.031382 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5153fe-67644d745-stq6l"] Nov 02 23:16:41.032443 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:41.032399 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-5153fe-67644d745-stq6l" podUID="fd8121c0-31b6-40d5-be58-3ada3872408c" containerName="payload-processing" containerID="cri-o://8e6c27c04988c37543a0b570bfea6103eec6552492d59de462eae620310f21e9" gracePeriod=25 Nov 02 23:16:41.405242 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:41.405056 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5153fe-67644d745-stq6l" Nov 02 23:16:41.478088 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:41.478057 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd8121c0-31b6-40d5-be58-3ada3872408c-kube-api-access-vg76k\" (UniqueName: \"kubernetes.io/projected/fd8121c0-31b6-40d5-be58-3ada3872408c-kube-api-access-vg76k\") pod \"fd8121c0-31b6-40d5-be58-3ada3872408c\" (UID: \"fd8121c0-31b6-40d5-be58-3ada3872408c\") " Nov 02 23:16:41.478275 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:41.478120 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd8121c0-31b6-40d5-be58-3ada3872408c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd8121c0-31b6-40d5-be58-3ada3872408c-plugins-config-volume\") pod \"fd8121c0-31b6-40d5-be58-3ada3872408c\" (UID: \"fd8121c0-31b6-40d5-be58-3ada3872408c\") " Nov 02 23:16:41.478480 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:41.478456 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd8121c0-31b6-40d5-be58-3ada3872408c-plugins-config-volume" pod "fd8121c0-31b6-40d5-be58-3ada3872408c" (UID: "fd8121c0-31b6-40d5-be58-3ada3872408c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:16:41.478724 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:41.478708 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd8121c0-31b6-40d5-be58-3ada3872408c-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:41.480639 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:41.480612 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8121c0-31b6-40d5-be58-3ada3872408c-kube-api-access-vg76k" pod "fd8121c0-31b6-40d5-be58-3ada3872408c" (UID: "fd8121c0-31b6-40d5-be58-3ada3872408c"). InnerVolumeSpecName "kube-api-access-vg76k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:16:41.580187 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:41.580118 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg76k\" (UniqueName: \"kubernetes.io/projected/fd8121c0-31b6-40d5-be58-3ada3872408c-kube-api-access-vg76k\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:42.017783 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:42.017735 5727 generic.go:356] "Generic (PLEG): container finished" podID="fd8121c0-31b6-40d5-be58-3ada3872408c" containerID="8e6c27c04988c37543a0b570bfea6103eec6552492d59de462eae620310f21e9" exitCode=0 Nov 02 23:16:42.017985 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:42.017937 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5153fe-67644d745-stq6l" event={"ID":"fd8121c0-31b6-40d5-be58-3ada3872408c","Type":"ContainerDied","Data":"8e6c27c04988c37543a0b570bfea6103eec6552492d59de462eae620310f21e9"} Nov 02 23:16:42.018047 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:42.018032 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5153fe-67644d745-stq6l" event={"ID":"fd8121c0-31b6-40d5-be58-3ada3872408c","Type":"ContainerDied","Data":"3bd43ea21577436b0e2c127ce5402a9ff1e355079d82feda54052d12ca019136"} Nov 02 23:16:42.018098 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:42.018071 5727 scope.go:122] "RemoveContainer" containerID="8e6c27c04988c37543a0b570bfea6103eec6552492d59de462eae620310f21e9" Nov 02 23:16:42.018303 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:42.018284 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5153fe-67644d745-stq6l" Nov 02 23:16:42.043557 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:42.041789 5727 scope.go:122] "RemoveContainer" containerID="8e6c27c04988c37543a0b570bfea6103eec6552492d59de462eae620310f21e9" Nov 02 23:16:42.043557 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:16:42.042393 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e6c27c04988c37543a0b570bfea6103eec6552492d59de462eae620310f21e9\": container with ID starting with 8e6c27c04988c37543a0b570bfea6103eec6552492d59de462eae620310f21e9 not found: ID does not exist" containerID="8e6c27c04988c37543a0b570bfea6103eec6552492d59de462eae620310f21e9" Nov 02 23:16:42.043557 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:42.042429 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e6c27c04988c37543a0b570bfea6103eec6552492d59de462eae620310f21e9"} err="failed to get container status \"8e6c27c04988c37543a0b570bfea6103eec6552492d59de462eae620310f21e9\": rpc error: code = NotFound desc = could not find container \"8e6c27c04988c37543a0b570bfea6103eec6552492d59de462eae620310f21e9\": container with ID starting with 8e6c27c04988c37543a0b570bfea6103eec6552492d59de462eae620310f21e9 not found: ID does not exist" Nov 02 23:16:42.184158 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:42.184054 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5153fe-67644d745-stq6l"] Nov 02 23:16:42.234883 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:42.234850 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5153fe-67644d745-stq6l"] Nov 02 23:16:43.885113 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:43.885078 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd8121c0-31b6-40d5-be58-3ada3872408c" path="/var/lib/kubelet/pods/fd8121c0-31b6-40d5-be58-3ada3872408c/volumes" Nov 02 23:16:57.692358 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:57.692325 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv"] Nov 02 23:16:57.692981 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:57.692967 5727 cpu_manager.go:414] "containerMap: removing container" podUID="57d13cfb-07f6-4709-8ceb-068e0528e721" containerName="istio-proxy" Nov 02 23:16:57.693027 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:57.692982 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57d13cfb-07f6-4709-8ceb-068e0528e721" containerName="istio-proxy" Nov 02 23:16:57.693027 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:57.692992 5727 cpu_manager.go:414] "containerMap: removing container" podUID="875c0021-b1a3-45b3-81a4-da889bf77c04" containerName="payload-pre-processing" Nov 02 23:16:57.693027 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:57.692997 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="875c0021-b1a3-45b3-81a4-da889bf77c04" containerName="payload-pre-processing" Nov 02 23:16:57.693027 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:57.693018 5727 cpu_manager.go:414] "containerMap: removing container" podUID="9be2be64-8a5a-4275-a973-1a78b432c62f" containerName="maas-api" Nov 02 23:16:57.693027 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:57.693022 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9be2be64-8a5a-4275-a973-1a78b432c62f" containerName="maas-api" Nov 02 23:16:57.693228 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:57.693035 5727 cpu_manager.go:414] "containerMap: removing container" podUID="fd8121c0-31b6-40d5-be58-3ada3872408c" containerName="payload-processing" Nov 02 23:16:57.693228 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:57.693041 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd8121c0-31b6-40d5-be58-3ada3872408c" containerName="payload-processing" Nov 02 23:16:57.693228 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:57.693075 5727 cpu_manager.go:414] "containerMap: removing container" podUID="af82c965-7e41-4e03-97fb-f84c11f0b73e" containerName="maas-api" Nov 02 23:16:57.693228 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:57.693083 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af82c965-7e41-4e03-97fb-f84c11f0b73e" containerName="maas-api" Nov 02 23:16:57.693228 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:57.693212 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="9be2be64-8a5a-4275-a973-1a78b432c62f" containerName="maas-api" Nov 02 23:16:57.693228 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:57.693221 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd8121c0-31b6-40d5-be58-3ada3872408c" containerName="payload-processing" Nov 02 23:16:57.693228 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:57.693230 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="af82c965-7e41-4e03-97fb-f84c11f0b73e" containerName="maas-api" Nov 02 23:16:57.693427 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:57.693242 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="57d13cfb-07f6-4709-8ceb-068e0528e721" containerName="istio-proxy" Nov 02 23:16:57.693427 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:57.693248 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="875c0021-b1a3-45b3-81a4-da889bf77c04" containerName="payload-pre-processing" Nov 02 23:16:57.696784 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:57.696768 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv" Nov 02 23:16:57.750523 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:57.750478 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a7a67875-serving-cert\"" Nov 02 23:16:57.796223 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:57.796193 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2e0f11e-4b44-4088-b095-7c9e47a07707-metrics-certs\") pod \"maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv\" (UID: \"c2e0f11e-4b44-4088-b095-7c9e47a07707\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv" Nov 02 23:16:57.796360 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:57.796233 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e0f11e-4b44-4088-b095-7c9e47a07707-openshift-service-ca\") pod \"maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv\" (UID: \"c2e0f11e-4b44-4088-b095-7c9e47a07707\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv" Nov 02 23:16:57.796360 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:57.796315 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e0f11e-4b44-4088-b095-7c9e47a07707-maas-api-tls\") pod \"maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv\" (UID: \"c2e0f11e-4b44-4088-b095-7c9e47a07707\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv" Nov 02 23:16:57.796452 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:57.796439 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skfvf\" (UniqueName: \"kubernetes.io/projected/c2e0f11e-4b44-4088-b095-7c9e47a07707-kube-api-access-skfvf\") pod \"maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv\" (UID: \"c2e0f11e-4b44-4088-b095-7c9e47a07707\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv" Nov 02 23:16:57.803939 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:57.803918 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv"] Nov 02 23:16:57.852018 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:16:57.851984 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:16:57.897895 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:57.897870 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2e0f11e-4b44-4088-b095-7c9e47a07707-metrics-certs\") pod \"maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv\" (UID: \"c2e0f11e-4b44-4088-b095-7c9e47a07707\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv" Nov 02 23:16:57.898085 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:57.897901 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e0f11e-4b44-4088-b095-7c9e47a07707-openshift-service-ca\") pod \"maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv\" (UID: \"c2e0f11e-4b44-4088-b095-7c9e47a07707\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv" Nov 02 23:16:57.898085 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:57.897948 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e0f11e-4b44-4088-b095-7c9e47a07707-maas-api-tls\") pod \"maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv\" (UID: \"c2e0f11e-4b44-4088-b095-7c9e47a07707\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv" Nov 02 23:16:57.898085 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:16:57.898047 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-serving-cert: secret "maas-api-e2e-mt-a7a67875-serving-cert" not found Nov 02 23:16:57.898310 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:16:57.898118 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2e0f11e-4b44-4088-b095-7c9e47a07707-maas-api-tls podName:c2e0f11e-4b44-4088-b095-7c9e47a07707 nodeName:}" failed. No retries permitted until 2026-11-02 23:16:58.398097165 +0000 UTC m=+1704.636594824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c2e0f11e-4b44-4088-b095-7c9e47a07707-maas-api-tls") pod "maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv" (UID: "c2e0f11e-4b44-4088-b095-7c9e47a07707") : secret "maas-api-e2e-mt-a7a67875-serving-cert" not found Nov 02 23:16:57.898382 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:57.898345 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skfvf\" (UniqueName: \"kubernetes.io/projected/c2e0f11e-4b44-4088-b095-7c9e47a07707-kube-api-access-skfvf\") pod \"maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv\" (UID: \"c2e0f11e-4b44-4088-b095-7c9e47a07707\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv" Nov 02 23:16:57.898796 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:57.898770 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e0f11e-4b44-4088-b095-7c9e47a07707-openshift-service-ca\") pod \"maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv\" (UID: \"c2e0f11e-4b44-4088-b095-7c9e47a07707\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv" Nov 02 23:16:57.900455 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:57.900435 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2e0f11e-4b44-4088-b095-7c9e47a07707-metrics-certs\") pod \"maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv\" (UID: \"c2e0f11e-4b44-4088-b095-7c9e47a07707\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv" Nov 02 23:16:57.960284 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:57.960194 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skfvf\" (UniqueName: \"kubernetes.io/projected/c2e0f11e-4b44-4088-b095-7c9e47a07707-kube-api-access-skfvf\") pod \"maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv\" (UID: \"c2e0f11e-4b44-4088-b095-7c9e47a07707\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv" Nov 02 23:16:58.413206 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:58.413176 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e0f11e-4b44-4088-b095-7c9e47a07707-maas-api-tls\") pod \"maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv\" (UID: \"c2e0f11e-4b44-4088-b095-7c9e47a07707\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv" Nov 02 23:16:58.415894 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:58.415868 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e0f11e-4b44-4088-b095-7c9e47a07707-maas-api-tls\") pod \"maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv\" (UID: \"c2e0f11e-4b44-4088-b095-7c9e47a07707\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv" Nov 02 23:16:58.612680 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:58.612649 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv" Nov 02 23:16:58.997288 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:58.997256 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv"] Nov 02 23:16:58.998867 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:16:58.998839 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2e0f11e_4b44_4088_b095_7c9e47a07707.slice/crio-cb8bfb79dda3b7aaa6cb6a34cd7f5256c3dd51405f9785a492bac655307d9b7c WatchSource:0}: Error finding container cb8bfb79dda3b7aaa6cb6a34cd7f5256c3dd51405f9785a492bac655307d9b7c: Status 404 returned error can't find the container with id cb8bfb79dda3b7aaa6cb6a34cd7f5256c3dd51405f9785a492bac655307d9b7c Nov 02 23:16:59.000734 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:59.000714 5727 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 23:16:59.161377 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:16:59.161341 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv" event={"ID":"c2e0f11e-4b44-4088-b095-7c9e47a07707","Type":"ContainerStarted","Data":"cb8bfb79dda3b7aaa6cb6a34cd7f5256c3dd51405f9785a492bac655307d9b7c"} Nov 02 23:17:00.173085 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:00.172992 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv" event={"ID":"c2e0f11e-4b44-4088-b095-7c9e47a07707","Type":"ContainerStarted","Data":"961888e46073aaf0ade78eae180b6a5d67483fb2ea55a31bc6f6c03998c8180b"} Nov 02 23:17:00.173571 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:00.173206 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv" Nov 02 23:17:00.332391 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:00.332341 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv" podStartSLOduration=2.456696246 podStartE2EDuration="3.332326322s" totalImagesPullingTime="875.630076ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:16:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:17:00.328601319 +0000 UTC m=+1706.567098997" watchObservedRunningTime="2026-11-02 23:17:00.332326322 +0000 UTC m=+1706.570824005" Nov 02 23:17:02.372481 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:02.372444 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2"] Nov 02 23:17:02.373006 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:02.372687 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2" podUID="e7f2af2d-71e7-42c8-a05e-86379dd05a44" containerName="maas-api" containerID="cri-o://fa0f61ac49d30da31737161fcc9c29fe441f0b176c6d0964a3bf722cc36452b7" gracePeriod=30 Nov 02 23:17:02.710989 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:02.710962 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2" Nov 02 23:17:02.774820 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:02.774696 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7f2af2d-71e7-42c8-a05e-86379dd05a44-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f2af2d-71e7-42c8-a05e-86379dd05a44-openshift-service-ca\") pod \"e7f2af2d-71e7-42c8-a05e-86379dd05a44\" (UID: \"e7f2af2d-71e7-42c8-a05e-86379dd05a44\") " Nov 02 23:17:02.774820 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:02.774754 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7f2af2d-71e7-42c8-a05e-86379dd05a44-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7f2af2d-71e7-42c8-a05e-86379dd05a44-maas-api-tls\") pod \"e7f2af2d-71e7-42c8-a05e-86379dd05a44\" (UID: \"e7f2af2d-71e7-42c8-a05e-86379dd05a44\") " Nov 02 23:17:02.775139 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:02.775001 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7f2af2d-71e7-42c8-a05e-86379dd05a44-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f2af2d-71e7-42c8-a05e-86379dd05a44-metrics-certs\") pod \"e7f2af2d-71e7-42c8-a05e-86379dd05a44\" (UID: \"e7f2af2d-71e7-42c8-a05e-86379dd05a44\") " Nov 02 23:17:02.775139 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:02.775045 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f2af2d-71e7-42c8-a05e-86379dd05a44-kube-api-access-jnf6n\" (UniqueName: \"kubernetes.io/projected/e7f2af2d-71e7-42c8-a05e-86379dd05a44-kube-api-access-jnf6n\") pod \"e7f2af2d-71e7-42c8-a05e-86379dd05a44\" (UID: \"e7f2af2d-71e7-42c8-a05e-86379dd05a44\") " Nov 02 23:17:02.776906 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:02.776723 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f2af2d-71e7-42c8-a05e-86379dd05a44-openshift-service-ca" pod "e7f2af2d-71e7-42c8-a05e-86379dd05a44" (UID: "e7f2af2d-71e7-42c8-a05e-86379dd05a44"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:17:02.784047 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:02.782625 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7f2af2d-71e7-42c8-a05e-86379dd05a44-maas-api-tls" pod "e7f2af2d-71e7-42c8-a05e-86379dd05a44" (UID: "e7f2af2d-71e7-42c8-a05e-86379dd05a44"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:17:02.787221 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:02.784503 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7f2af2d-71e7-42c8-a05e-86379dd05a44-metrics-certs" pod "e7f2af2d-71e7-42c8-a05e-86379dd05a44" (UID: "e7f2af2d-71e7-42c8-a05e-86379dd05a44"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:17:02.788755 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:02.788722 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f2af2d-71e7-42c8-a05e-86379dd05a44-kube-api-access-jnf6n" pod "e7f2af2d-71e7-42c8-a05e-86379dd05a44" (UID: "e7f2af2d-71e7-42c8-a05e-86379dd05a44"). InnerVolumeSpecName "kube-api-access-jnf6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:17:02.876684 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:02.876649 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f2af2d-71e7-42c8-a05e-86379dd05a44-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:02.876684 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:02.876688 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnf6n\" (UniqueName: \"kubernetes.io/projected/e7f2af2d-71e7-42c8-a05e-86379dd05a44-kube-api-access-jnf6n\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:02.876941 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:02.876705 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f2af2d-71e7-42c8-a05e-86379dd05a44-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:02.876941 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:02.876717 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7f2af2d-71e7-42c8-a05e-86379dd05a44-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:03.198258 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:03.198221 5727 generic.go:356] "Generic (PLEG): container finished" podID="e7f2af2d-71e7-42c8-a05e-86379dd05a44" containerID="fa0f61ac49d30da31737161fcc9c29fe441f0b176c6d0964a3bf722cc36452b7" exitCode=0 Nov 02 23:17:03.198412 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:03.198281 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2" Nov 02 23:17:03.198412 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:03.198312 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2" event={"ID":"e7f2af2d-71e7-42c8-a05e-86379dd05a44","Type":"ContainerDied","Data":"fa0f61ac49d30da31737161fcc9c29fe441f0b176c6d0964a3bf722cc36452b7"} Nov 02 23:17:03.198412 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:03.198369 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2" event={"ID":"e7f2af2d-71e7-42c8-a05e-86379dd05a44","Type":"ContainerDied","Data":"bc55501f082530f513b98bf940360399ff31bfe2d71e2772a8d0872ae509962c"} Nov 02 23:17:03.198412 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:03.198393 5727 scope.go:122] "RemoveContainer" containerID="fa0f61ac49d30da31737161fcc9c29fe441f0b176c6d0964a3bf722cc36452b7" Nov 02 23:17:03.214532 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:03.214504 5727 scope.go:122] "RemoveContainer" containerID="fa0f61ac49d30da31737161fcc9c29fe441f0b176c6d0964a3bf722cc36452b7" Nov 02 23:17:03.214806 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:17:03.214780 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa0f61ac49d30da31737161fcc9c29fe441f0b176c6d0964a3bf722cc36452b7\": container with ID starting with fa0f61ac49d30da31737161fcc9c29fe441f0b176c6d0964a3bf722cc36452b7 not found: ID does not exist" containerID="fa0f61ac49d30da31737161fcc9c29fe441f0b176c6d0964a3bf722cc36452b7" Nov 02 23:17:03.214874 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:03.214814 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa0f61ac49d30da31737161fcc9c29fe441f0b176c6d0964a3bf722cc36452b7"} err="failed to get container status \"fa0f61ac49d30da31737161fcc9c29fe441f0b176c6d0964a3bf722cc36452b7\": rpc error: code = NotFound desc = could not find container \"fa0f61ac49d30da31737161fcc9c29fe441f0b176c6d0964a3bf722cc36452b7\": container with ID starting with fa0f61ac49d30da31737161fcc9c29fe441f0b176c6d0964a3bf722cc36452b7 not found: ID does not exist" Nov 02 23:17:03.373245 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:03.373213 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2"] Nov 02 23:17:03.375354 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:03.375225 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5153fe-6ff67598b6-9h9n2"] Nov 02 23:17:03.532446 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:03.532360 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8ab111-799db7457d-4c4nn"] Nov 02 23:17:03.532626 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:03.532548 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8ab111-799db7457d-4c4nn" podUID="9dce87c6-bbea-42af-8ac5-0655d4b4d352" containerName="maas-api" containerID="cri-o://4fd356862564d8b155f7244d402c0459a662b7678c8482aabed5527428fe236d" gracePeriod=30 Nov 02 23:17:03.839903 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:03.839880 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8ab111-799db7457d-4c4nn" Nov 02 23:17:03.884255 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:03.884226 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7f2af2d-71e7-42c8-a05e-86379dd05a44" path="/var/lib/kubelet/pods/e7f2af2d-71e7-42c8-a05e-86379dd05a44/volumes" Nov 02 23:17:03.899964 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:03.899934 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dce87c6-bbea-42af-8ac5-0655d4b4d352-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dce87c6-bbea-42af-8ac5-0655d4b4d352-openshift-service-ca\") pod \"9dce87c6-bbea-42af-8ac5-0655d4b4d352\" (UID: \"9dce87c6-bbea-42af-8ac5-0655d4b4d352\") " Nov 02 23:17:03.900103 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:03.900035 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dce87c6-bbea-42af-8ac5-0655d4b4d352-kube-api-access-pt9xw\" (UniqueName: \"kubernetes.io/projected/9dce87c6-bbea-42af-8ac5-0655d4b4d352-kube-api-access-pt9xw\") pod \"9dce87c6-bbea-42af-8ac5-0655d4b4d352\" (UID: \"9dce87c6-bbea-42af-8ac5-0655d4b4d352\") " Nov 02 23:17:03.900219 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:03.900197 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dce87c6-bbea-42af-8ac5-0655d4b4d352-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dce87c6-bbea-42af-8ac5-0655d4b4d352-metrics-certs\") pod \"9dce87c6-bbea-42af-8ac5-0655d4b4d352\" (UID: \"9dce87c6-bbea-42af-8ac5-0655d4b4d352\") " Nov 02 23:17:03.900284 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:03.900263 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dce87c6-bbea-42af-8ac5-0655d4b4d352-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dce87c6-bbea-42af-8ac5-0655d4b4d352-maas-api-tls\") pod \"9dce87c6-bbea-42af-8ac5-0655d4b4d352\" (UID: \"9dce87c6-bbea-42af-8ac5-0655d4b4d352\") " Nov 02 23:17:03.900351 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:03.900312 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dce87c6-bbea-42af-8ac5-0655d4b4d352-openshift-service-ca" pod "9dce87c6-bbea-42af-8ac5-0655d4b4d352" (UID: "9dce87c6-bbea-42af-8ac5-0655d4b4d352"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:17:03.900744 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:03.900719 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dce87c6-bbea-42af-8ac5-0655d4b4d352-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:03.902408 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:03.902384 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dce87c6-bbea-42af-8ac5-0655d4b4d352-kube-api-access-pt9xw" pod "9dce87c6-bbea-42af-8ac5-0655d4b4d352" (UID: "9dce87c6-bbea-42af-8ac5-0655d4b4d352"). InnerVolumeSpecName "kube-api-access-pt9xw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:17:03.902507 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:03.902470 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dce87c6-bbea-42af-8ac5-0655d4b4d352-maas-api-tls" pod "9dce87c6-bbea-42af-8ac5-0655d4b4d352" (UID: "9dce87c6-bbea-42af-8ac5-0655d4b4d352"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:17:03.902507 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:03.902494 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dce87c6-bbea-42af-8ac5-0655d4b4d352-metrics-certs" pod "9dce87c6-bbea-42af-8ac5-0655d4b4d352" (UID: "9dce87c6-bbea-42af-8ac5-0655d4b4d352"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:17:04.005190 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:04.002583 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt9xw\" (UniqueName: \"kubernetes.io/projected/9dce87c6-bbea-42af-8ac5-0655d4b4d352-kube-api-access-pt9xw\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:04.005190 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:04.002625 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dce87c6-bbea-42af-8ac5-0655d4b4d352-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:04.005190 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:04.002642 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dce87c6-bbea-42af-8ac5-0655d4b4d352-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:04.209307 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:04.209268 5727 generic.go:356] "Generic (PLEG): container finished" podID="9dce87c6-bbea-42af-8ac5-0655d4b4d352" containerID="4fd356862564d8b155f7244d402c0459a662b7678c8482aabed5527428fe236d" exitCode=0 Nov 02 23:17:04.209486 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:04.209331 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8ab111-799db7457d-4c4nn" Nov 02 23:17:04.209486 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:04.209359 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8ab111-799db7457d-4c4nn" event={"ID":"9dce87c6-bbea-42af-8ac5-0655d4b4d352","Type":"ContainerDied","Data":"4fd356862564d8b155f7244d402c0459a662b7678c8482aabed5527428fe236d"} Nov 02 23:17:04.209486 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:04.209415 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8ab111-799db7457d-4c4nn" event={"ID":"9dce87c6-bbea-42af-8ac5-0655d4b4d352","Type":"ContainerDied","Data":"eacbe46430eb23ca98cf8faf4f49a77adcf6a630fccd4b5d171ed0a9eef56816"} Nov 02 23:17:04.209486 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:04.209442 5727 scope.go:122] "RemoveContainer" containerID="4fd356862564d8b155f7244d402c0459a662b7678c8482aabed5527428fe236d" Nov 02 23:17:04.221619 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:04.221604 5727 scope.go:122] "RemoveContainer" containerID="4fd356862564d8b155f7244d402c0459a662b7678c8482aabed5527428fe236d" Nov 02 23:17:04.221868 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:17:04.221849 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd356862564d8b155f7244d402c0459a662b7678c8482aabed5527428fe236d\": container with ID starting with 4fd356862564d8b155f7244d402c0459a662b7678c8482aabed5527428fe236d not found: ID does not exist" containerID="4fd356862564d8b155f7244d402c0459a662b7678c8482aabed5527428fe236d" Nov 02 23:17:04.221921 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:04.221877 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd356862564d8b155f7244d402c0459a662b7678c8482aabed5527428fe236d"} err="failed to get container status \"4fd356862564d8b155f7244d402c0459a662b7678c8482aabed5527428fe236d\": rpc error: code = NotFound desc = could not find container \"4fd356862564d8b155f7244d402c0459a662b7678c8482aabed5527428fe236d\": container with ID starting with 4fd356862564d8b155f7244d402c0459a662b7678c8482aabed5527428fe236d not found: ID does not exist" Nov 02 23:17:04.378217 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:04.377091 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8ab111-799db7457d-4c4nn"] Nov 02 23:17:04.382752 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:04.382724 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8ab111-799db7457d-4c4nn"] Nov 02 23:17:05.887444 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:05.887376 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dce87c6-bbea-42af-8ac5-0655d4b4d352" path="/var/lib/kubelet/pods/9dce87c6-bbea-42af-8ac5-0655d4b4d352/volumes" Nov 02 23:17:06.186537 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:06.186451 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv" Nov 02 23:17:26.898099 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:26.898061 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc"] Nov 02 23:17:26.898789 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:26.898772 5727 cpu_manager.go:414] "containerMap: removing container" podUID="9dce87c6-bbea-42af-8ac5-0655d4b4d352" containerName="maas-api" Nov 02 23:17:26.898832 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:26.898791 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dce87c6-bbea-42af-8ac5-0655d4b4d352" containerName="maas-api" Nov 02 23:17:26.898832 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:26.898816 5727 cpu_manager.go:414] "containerMap: removing container" podUID="e7f2af2d-71e7-42c8-a05e-86379dd05a44" containerName="maas-api" Nov 02 23:17:26.898832 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:26.898821 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f2af2d-71e7-42c8-a05e-86379dd05a44" containerName="maas-api" Nov 02 23:17:26.898924 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:26.898909 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dce87c6-bbea-42af-8ac5-0655d4b4d352" containerName="maas-api" Nov 02 23:17:26.898924 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:26.898920 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f2af2d-71e7-42c8-a05e-86379dd05a44" containerName="maas-api" Nov 02 23:17:26.902490 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:26.902471 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc" Nov 02 23:17:26.955328 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:26.955292 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-aa2454-serving-cert\"" Nov 02 23:17:27.009057 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:27.009023 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc"] Nov 02 23:17:27.048005 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:27.047968 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98dn5\" (UniqueName: \"kubernetes.io/projected/19c83f05-c0d9-47f8-a8ab-5cf565015621-kube-api-access-98dn5\") pod \"maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc\" (UID: \"19c83f05-c0d9-47f8-a8ab-5cf565015621\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc" Nov 02 23:17:27.048213 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:27.048034 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19c83f05-c0d9-47f8-a8ab-5cf565015621-metrics-certs\") pod \"maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc\" (UID: \"19c83f05-c0d9-47f8-a8ab-5cf565015621\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc" Nov 02 23:17:27.048270 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:27.048209 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19c83f05-c0d9-47f8-a8ab-5cf565015621-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc\" (UID: \"19c83f05-c0d9-47f8-a8ab-5cf565015621\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc" Nov 02 23:17:27.048397 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:27.048379 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19c83f05-c0d9-47f8-a8ab-5cf565015621-maas-api-tls\") pod \"maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc\" (UID: \"19c83f05-c0d9-47f8-a8ab-5cf565015621\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc" Nov 02 23:17:27.150344 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:27.150248 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19c83f05-c0d9-47f8-a8ab-5cf565015621-maas-api-tls\") pod \"maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc\" (UID: \"19c83f05-c0d9-47f8-a8ab-5cf565015621\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc" Nov 02 23:17:27.150506 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:17:27.150428 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa2454-serving-cert: secret "maas-api-e2e-ipp-a-aa2454-serving-cert" not found Nov 02 23:17:27.150506 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:27.150454 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98dn5\" (UniqueName: \"kubernetes.io/projected/19c83f05-c0d9-47f8-a8ab-5cf565015621-kube-api-access-98dn5\") pod \"maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc\" (UID: \"19c83f05-c0d9-47f8-a8ab-5cf565015621\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc" Nov 02 23:17:27.150628 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:27.150502 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19c83f05-c0d9-47f8-a8ab-5cf565015621-metrics-certs\") pod \"maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc\" (UID: \"19c83f05-c0d9-47f8-a8ab-5cf565015621\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc" Nov 02 23:17:27.150628 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:27.150576 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19c83f05-c0d9-47f8-a8ab-5cf565015621-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc\" (UID: \"19c83f05-c0d9-47f8-a8ab-5cf565015621\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc" Nov 02 23:17:27.150827 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:17:27.150804 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19c83f05-c0d9-47f8-a8ab-5cf565015621-maas-api-tls podName:19c83f05-c0d9-47f8-a8ab-5cf565015621 nodeName:}" failed. No retries permitted until 2026-11-02 23:17:27.650781605 +0000 UTC m=+1733.889279246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/19c83f05-c0d9-47f8-a8ab-5cf565015621-maas-api-tls") pod "maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc" (UID: "19c83f05-c0d9-47f8-a8ab-5cf565015621") : secret "maas-api-e2e-ipp-a-aa2454-serving-cert" not found Nov 02 23:17:27.151540 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:27.151503 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19c83f05-c0d9-47f8-a8ab-5cf565015621-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc\" (UID: \"19c83f05-c0d9-47f8-a8ab-5cf565015621\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc" Nov 02 23:17:27.153637 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:27.153612 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19c83f05-c0d9-47f8-a8ab-5cf565015621-metrics-certs\") pod \"maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc\" (UID: \"19c83f05-c0d9-47f8-a8ab-5cf565015621\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc" Nov 02 23:17:27.213662 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:27.213629 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98dn5\" (UniqueName: \"kubernetes.io/projected/19c83f05-c0d9-47f8-a8ab-5cf565015621-kube-api-access-98dn5\") pod \"maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc\" (UID: \"19c83f05-c0d9-47f8-a8ab-5cf565015621\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc" Nov 02 23:17:27.658531 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:27.658489 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19c83f05-c0d9-47f8-a8ab-5cf565015621-maas-api-tls\") pod \"maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc\" (UID: \"19c83f05-c0d9-47f8-a8ab-5cf565015621\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc" Nov 02 23:17:27.661172 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:27.661147 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19c83f05-c0d9-47f8-a8ab-5cf565015621-maas-api-tls\") pod \"maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc\" (UID: \"19c83f05-c0d9-47f8-a8ab-5cf565015621\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc" Nov 02 23:17:27.818191 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:27.818118 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc" Nov 02 23:17:27.852623 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:17:27.852584 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:17:28.209220 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:17:28.209185 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c83f05_c0d9_47f8_a8ab_5cf565015621.slice/crio-0ad39fd8ec514a15d901ace24fe5ab3834fb40c00c88d070cf394518ac5ac5e0 WatchSource:0}: Error finding container 0ad39fd8ec514a15d901ace24fe5ab3834fb40c00c88d070cf394518ac5ac5e0: Status 404 returned error can't find the container with id 0ad39fd8ec514a15d901ace24fe5ab3834fb40c00c88d070cf394518ac5ac5e0 Nov 02 23:17:28.210342 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:28.210304 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc"] Nov 02 23:17:28.444349 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:28.444307 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc" event={"ID":"19c83f05-c0d9-47f8-a8ab-5cf565015621","Type":"ContainerStarted","Data":"0ad39fd8ec514a15d901ace24fe5ab3834fb40c00c88d070cf394518ac5ac5e0"} Nov 02 23:17:29.454007 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:29.453834 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc" event={"ID":"19c83f05-c0d9-47f8-a8ab-5cf565015621","Type":"ContainerStarted","Data":"824526bfec4f627dc093d6558f6dd3a03cc5849bd3d7de7482644a80c416b842"} Nov 02 23:17:29.454007 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:29.453926 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc" Nov 02 23:17:29.576370 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:29.576305 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc" podStartSLOduration=2.799404485 podStartE2EDuration="3.576286874s" totalImagesPullingTime="776.882389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:17:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:17:29.571321628 +0000 UTC m=+1735.809819290" watchObservedRunningTime="2026-11-02 23:17:29.576286874 +0000 UTC m=+1735.814784537" Nov 02 23:17:35.470763 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:35.470734 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc" Nov 02 23:17:36.900588 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:36.900556 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl"] Nov 02 23:17:36.907056 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:36.907034 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl" Nov 02 23:17:36.967163 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:36.967103 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl"] Nov 02 23:17:36.984180 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:36.984152 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh6jg\" (UniqueName: \"kubernetes.io/projected/d1a789e5-5998-4ad8-9198-7dee53101d6d-kube-api-access-rh6jg\") pod \"maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl\" (UID: \"d1a789e5-5998-4ad8-9198-7dee53101d6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl" Nov 02 23:17:36.984336 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:36.984191 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1a789e5-5998-4ad8-9198-7dee53101d6d-metrics-certs\") pod \"maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl\" (UID: \"d1a789e5-5998-4ad8-9198-7dee53101d6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl" Nov 02 23:17:36.984385 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:36.984347 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1a789e5-5998-4ad8-9198-7dee53101d6d-openshift-service-ca\") pod \"maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl\" (UID: \"d1a789e5-5998-4ad8-9198-7dee53101d6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl" Nov 02 23:17:36.984441 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:36.984400 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1a789e5-5998-4ad8-9198-7dee53101d6d-maas-api-tls\") pod \"maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl\" (UID: \"d1a789e5-5998-4ad8-9198-7dee53101d6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl" Nov 02 23:17:37.008102 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:37.008073 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-048ef3-serving-cert\"" Nov 02 23:17:37.085874 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:37.085834 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh6jg\" (UniqueName: \"kubernetes.io/projected/d1a789e5-5998-4ad8-9198-7dee53101d6d-kube-api-access-rh6jg\") pod \"maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl\" (UID: \"d1a789e5-5998-4ad8-9198-7dee53101d6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl" Nov 02 23:17:37.086089 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:37.085889 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1a789e5-5998-4ad8-9198-7dee53101d6d-metrics-certs\") pod \"maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl\" (UID: \"d1a789e5-5998-4ad8-9198-7dee53101d6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl" Nov 02 23:17:37.086089 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:37.085953 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1a789e5-5998-4ad8-9198-7dee53101d6d-openshift-service-ca\") pod \"maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl\" (UID: \"d1a789e5-5998-4ad8-9198-7dee53101d6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl" Nov 02 23:17:37.086089 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:37.085986 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1a789e5-5998-4ad8-9198-7dee53101d6d-maas-api-tls\") pod \"maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl\" (UID: \"d1a789e5-5998-4ad8-9198-7dee53101d6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl" Nov 02 23:17:37.086375 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:17:37.086168 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-048ef3-serving-cert: secret "maas-api-e2e-api-b-048ef3-serving-cert" not found Nov 02 23:17:37.086375 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:17:37.086280 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1a789e5-5998-4ad8-9198-7dee53101d6d-maas-api-tls podName:d1a789e5-5998-4ad8-9198-7dee53101d6d nodeName:}" failed. No retries permitted until 2026-11-02 23:17:37.586262136 +0000 UTC m=+1743.824759789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d1a789e5-5998-4ad8-9198-7dee53101d6d-maas-api-tls") pod "maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl" (UID: "d1a789e5-5998-4ad8-9198-7dee53101d6d") : secret "maas-api-e2e-api-b-048ef3-serving-cert" not found Nov 02 23:17:37.086843 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:37.086817 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1a789e5-5998-4ad8-9198-7dee53101d6d-openshift-service-ca\") pod \"maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl\" (UID: \"d1a789e5-5998-4ad8-9198-7dee53101d6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl" Nov 02 23:17:37.088841 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:37.088818 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1a789e5-5998-4ad8-9198-7dee53101d6d-metrics-certs\") pod \"maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl\" (UID: \"d1a789e5-5998-4ad8-9198-7dee53101d6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl" Nov 02 23:17:37.156259 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:37.156164 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh6jg\" (UniqueName: \"kubernetes.io/projected/d1a789e5-5998-4ad8-9198-7dee53101d6d-kube-api-access-rh6jg\") pod \"maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl\" (UID: \"d1a789e5-5998-4ad8-9198-7dee53101d6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl" Nov 02 23:17:37.593945 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:37.593910 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1a789e5-5998-4ad8-9198-7dee53101d6d-maas-api-tls\") pod \"maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl\" (UID: \"d1a789e5-5998-4ad8-9198-7dee53101d6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl" Nov 02 23:17:37.596685 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:37.596651 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1a789e5-5998-4ad8-9198-7dee53101d6d-maas-api-tls\") pod \"maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl\" (UID: \"d1a789e5-5998-4ad8-9198-7dee53101d6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl" Nov 02 23:17:37.821649 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:37.821614 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl" Nov 02 23:17:38.203728 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:38.203698 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl"] Nov 02 23:17:38.204640 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:17:38.204615 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a789e5_5998_4ad8_9198_7dee53101d6d.slice/crio-c00368316c6b25786db8d0b06ea4ee393f323d368d1219d975d2852926208d4b WatchSource:0}: Error finding container c00368316c6b25786db8d0b06ea4ee393f323d368d1219d975d2852926208d4b: Status 404 returned error can't find the container with id c00368316c6b25786db8d0b06ea4ee393f323d368d1219d975d2852926208d4b Nov 02 23:17:38.532584 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:38.532492 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl" event={"ID":"d1a789e5-5998-4ad8-9198-7dee53101d6d","Type":"ContainerStarted","Data":"c00368316c6b25786db8d0b06ea4ee393f323d368d1219d975d2852926208d4b"} Nov 02 23:17:39.543174 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:39.543118 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl" event={"ID":"d1a789e5-5998-4ad8-9198-7dee53101d6d","Type":"ContainerStarted","Data":"a12ba56632bb57ee040504633be8d517c095466d167c4bfd405f83e8c5f8303d"} Nov 02 23:17:39.543614 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:39.543225 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl" Nov 02 23:17:39.666099 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:39.666035 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl" podStartSLOduration=2.8714578189999997 podStartE2EDuration="3.666015962s" totalImagesPullingTime="794.558143ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:17:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:17:39.660706359 +0000 UTC m=+1745.899204022" watchObservedRunningTime="2026-11-02 23:17:39.666015962 +0000 UTC m=+1745.904513624" Nov 02 23:17:45.557780 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:17:45.557738 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl" Nov 02 23:17:57.846673 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:17:57.846629 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:18:07.086250 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:07.086218 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv"] Nov 02 23:18:07.086847 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:07.086413 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv" podUID="11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d" containerName="maas-api" containerID="cri-o://ea17c9f09251e27128588ac820409f69752d64cc29e8fd396c4c9c3cd521d3b4" gracePeriod=30 Nov 02 23:18:07.105656 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:07.105623 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv" podUID="11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.117:8443/health\": dial tcp 10.132.0.117:8443: connect: connection refused" Nov 02 23:18:07.378981 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:07.378950 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv" Nov 02 23:18:07.509981 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:07.508712 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d-kube-api-access-8jk77\" (UniqueName: \"kubernetes.io/projected/11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d-kube-api-access-8jk77\") pod \"11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d\" (UID: \"11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d\") " Nov 02 23:18:07.509981 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:07.508804 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d-openshift-service-ca\") pod \"11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d\" (UID: \"11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d\") " Nov 02 23:18:07.509981 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:07.508879 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d-metrics-certs\") pod \"11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d\" (UID: \"11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d\") " Nov 02 23:18:07.509981 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:07.508994 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d-maas-api-tls\") pod \"11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d\" (UID: \"11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d\") " Nov 02 23:18:07.509981 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:07.509516 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d-openshift-service-ca" pod "11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d" (UID: "11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:18:07.509981 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:07.509777 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:18:07.524698 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:07.524636 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d-maas-api-tls" pod "11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d" (UID: "11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:18:07.525189 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:07.525110 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d-metrics-certs" pod "11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d" (UID: "11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:18:07.526292 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:07.526261 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d-kube-api-access-8jk77" pod "11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d" (UID: "11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d"). InnerVolumeSpecName "kube-api-access-8jk77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:07.611464 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:07.611363 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jk77\" (UniqueName: \"kubernetes.io/projected/11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d-kube-api-access-8jk77\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:18:07.611464 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:07.611399 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:18:07.611464 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:07.611413 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:18:07.768168 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:07.768111 5727 generic.go:356] "Generic (PLEG): container finished" podID="11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d" containerID="ea17c9f09251e27128588ac820409f69752d64cc29e8fd396c4c9c3cd521d3b4" exitCode=0 Nov 02 23:18:07.768378 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:07.768184 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv" event={"ID":"11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d","Type":"ContainerDied","Data":"ea17c9f09251e27128588ac820409f69752d64cc29e8fd396c4c9c3cd521d3b4"} Nov 02 23:18:07.768378 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:07.768242 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv" event={"ID":"11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d","Type":"ContainerDied","Data":"e96d602d0d66749b9010267f1484f4aeaf0979f8e6cfbb0b34a27d1b4327e694"} Nov 02 23:18:07.768378 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:07.768193 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv" Nov 02 23:18:07.768378 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:07.768268 5727 scope.go:122] "RemoveContainer" containerID="ea17c9f09251e27128588ac820409f69752d64cc29e8fd396c4c9c3cd521d3b4" Nov 02 23:18:07.781567 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:07.781548 5727 scope.go:122] "RemoveContainer" containerID="ea17c9f09251e27128588ac820409f69752d64cc29e8fd396c4c9c3cd521d3b4" Nov 02 23:18:07.781902 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:18:07.781879 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea17c9f09251e27128588ac820409f69752d64cc29e8fd396c4c9c3cd521d3b4\": container with ID starting with ea17c9f09251e27128588ac820409f69752d64cc29e8fd396c4c9c3cd521d3b4 not found: ID does not exist" containerID="ea17c9f09251e27128588ac820409f69752d64cc29e8fd396c4c9c3cd521d3b4" Nov 02 23:18:07.782007 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:07.781910 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea17c9f09251e27128588ac820409f69752d64cc29e8fd396c4c9c3cd521d3b4"} err="failed to get container status \"ea17c9f09251e27128588ac820409f69752d64cc29e8fd396c4c9c3cd521d3b4\": rpc error: code = NotFound desc = could not find container \"ea17c9f09251e27128588ac820409f69752d64cc29e8fd396c4c9c3cd521d3b4\": container with ID starting with ea17c9f09251e27128588ac820409f69752d64cc29e8fd396c4c9c3cd521d3b4 not found: ID does not exist" Nov 02 23:18:07.933570 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:07.933483 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv"] Nov 02 23:18:07.983213 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:07.983186 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-62ab55-7d7d58989f-s5jmv"] Nov 02 23:18:09.885365 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:09.885322 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d" path="/var/lib/kubelet/pods/11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d/volumes" Nov 02 23:18:27.847703 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:18:27.847661 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:18:34.403059 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:34.403030 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 02 23:18:34.413814 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:34.413793 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 02 23:18:34.418356 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:34.418335 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 23:18:34.432355 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:34.432338 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 23:18:52.365631 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:52.365537 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bf51d6-8f6bfb495-bqqqx"] Nov 02 23:18:52.477200 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:52.477162 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bf51d6-6db79f996c-6c4dz"] Nov 02 23:18:57.366236 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:57.366177 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-bf51d6-8f6bfb495-bqqqx" podUID="6f7317cd-3f9f-4e50-a016-d18de7b5e8a6" containerName="payload-processing" containerID="cri-o://0468ea4cc006b47d022cb161a3f61aa38e7383dcea8e3b95f5d909809032662e" gracePeriod=25 Nov 02 23:18:57.477597 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:57.477510 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf51d6-6db79f996c-6c4dz" podUID="d0d6dc88-070f-4891-8e58-8adb3606cde0" containerName="payload-pre-processing" containerID="cri-o://8277f05dbbcf33945e862601f2786d15c7f9922dc52f06cc6be202f715f241b2" gracePeriod=25 Nov 02 23:18:57.676200 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:57.675906 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bf51d6-8f6bfb495-bqqqx" Nov 02 23:18:57.748311 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:57.748201 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f7317cd-3f9f-4e50-a016-d18de7b5e8a6-kube-api-access-cztk4\" (UniqueName: \"kubernetes.io/projected/6f7317cd-3f9f-4e50-a016-d18de7b5e8a6-kube-api-access-cztk4\") pod \"6f7317cd-3f9f-4e50-a016-d18de7b5e8a6\" (UID: \"6f7317cd-3f9f-4e50-a016-d18de7b5e8a6\") " Nov 02 23:18:57.748311 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:57.748275 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f7317cd-3f9f-4e50-a016-d18de7b5e8a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7317cd-3f9f-4e50-a016-d18de7b5e8a6-plugins-config-volume\") pod \"6f7317cd-3f9f-4e50-a016-d18de7b5e8a6\" (UID: \"6f7317cd-3f9f-4e50-a016-d18de7b5e8a6\") " Nov 02 23:18:57.749112 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:57.749012 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f7317cd-3f9f-4e50-a016-d18de7b5e8a6-plugins-config-volume" pod "6f7317cd-3f9f-4e50-a016-d18de7b5e8a6" (UID: "6f7317cd-3f9f-4e50-a016-d18de7b5e8a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:18:57.750899 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:57.750858 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f7317cd-3f9f-4e50-a016-d18de7b5e8a6-kube-api-access-cztk4" pod "6f7317cd-3f9f-4e50-a016-d18de7b5e8a6" (UID: "6f7317cd-3f9f-4e50-a016-d18de7b5e8a6"). InnerVolumeSpecName "kube-api-access-cztk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:57.782507 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:57.782477 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf51d6-6db79f996c-6c4dz" Nov 02 23:18:57.849701 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:18:57.849641 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:18:57.849864 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:57.849741 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0d6dc88-070f-4891-8e58-8adb3606cde0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0d6dc88-070f-4891-8e58-8adb3606cde0-plugins-config-volume\") pod \"d0d6dc88-070f-4891-8e58-8adb3606cde0\" (UID: \"d0d6dc88-070f-4891-8e58-8adb3606cde0\") " Nov 02 23:18:57.849945 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:57.849865 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0d6dc88-070f-4891-8e58-8adb3606cde0-kube-api-access-7m552\" (UniqueName: \"kubernetes.io/projected/d0d6dc88-070f-4891-8e58-8adb3606cde0-kube-api-access-7m552\") pod \"d0d6dc88-070f-4891-8e58-8adb3606cde0\" (UID: \"d0d6dc88-070f-4891-8e58-8adb3606cde0\") " Nov 02 23:18:57.850107 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:57.850085 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0d6dc88-070f-4891-8e58-8adb3606cde0-plugins-config-volume" pod "d0d6dc88-070f-4891-8e58-8adb3606cde0" (UID: "d0d6dc88-070f-4891-8e58-8adb3606cde0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:18:57.850495 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:57.850476 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0d6dc88-070f-4891-8e58-8adb3606cde0-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:18:57.850598 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:57.850501 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cztk4\" (UniqueName: \"kubernetes.io/projected/6f7317cd-3f9f-4e50-a016-d18de7b5e8a6-kube-api-access-cztk4\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:18:57.850598 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:57.850518 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7317cd-3f9f-4e50-a016-d18de7b5e8a6-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:18:57.852395 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:57.852345 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d6dc88-070f-4891-8e58-8adb3606cde0-kube-api-access-7m552" pod "d0d6dc88-070f-4891-8e58-8adb3606cde0" (UID: "d0d6dc88-070f-4891-8e58-8adb3606cde0"). InnerVolumeSpecName "kube-api-access-7m552". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:57.952170 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:57.952113 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m552\" (UniqueName: \"kubernetes.io/projected/d0d6dc88-070f-4891-8e58-8adb3606cde0-kube-api-access-7m552\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:18:58.185494 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:58.185462 5727 generic.go:356] "Generic (PLEG): container finished" podID="6f7317cd-3f9f-4e50-a016-d18de7b5e8a6" containerID="0468ea4cc006b47d022cb161a3f61aa38e7383dcea8e3b95f5d909809032662e" exitCode=0 Nov 02 23:18:58.185692 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:58.185521 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bf51d6-8f6bfb495-bqqqx" Nov 02 23:18:58.185692 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:58.185523 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bf51d6-8f6bfb495-bqqqx" event={"ID":"6f7317cd-3f9f-4e50-a016-d18de7b5e8a6","Type":"ContainerDied","Data":"0468ea4cc006b47d022cb161a3f61aa38e7383dcea8e3b95f5d909809032662e"} Nov 02 23:18:58.185692 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:58.185583 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bf51d6-8f6bfb495-bqqqx" event={"ID":"6f7317cd-3f9f-4e50-a016-d18de7b5e8a6","Type":"ContainerDied","Data":"c218e6ef9f720ccacd50d392fa781444e4e52e2dfd01a1bd9b821e6d5d8b0b7e"} Nov 02 23:18:58.185692 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:58.185611 5727 scope.go:122] "RemoveContainer" containerID="0468ea4cc006b47d022cb161a3f61aa38e7383dcea8e3b95f5d909809032662e" Nov 02 23:18:58.187573 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:58.187548 5727 generic.go:356] "Generic (PLEG): container finished" podID="d0d6dc88-070f-4891-8e58-8adb3606cde0" containerID="8277f05dbbcf33945e862601f2786d15c7f9922dc52f06cc6be202f715f241b2" exitCode=0 Nov 02 23:18:58.187699 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:58.187586 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf51d6-6db79f996c-6c4dz" event={"ID":"d0d6dc88-070f-4891-8e58-8adb3606cde0","Type":"ContainerDied","Data":"8277f05dbbcf33945e862601f2786d15c7f9922dc52f06cc6be202f715f241b2"} Nov 02 23:18:58.187699 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:58.187598 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf51d6-6db79f996c-6c4dz" Nov 02 23:18:58.187699 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:58.187626 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf51d6-6db79f996c-6c4dz" event={"ID":"d0d6dc88-070f-4891-8e58-8adb3606cde0","Type":"ContainerDied","Data":"bd88fb05238abb82553c9b351979a7709a3f620276307551413e80c77b2aa9d9"} Nov 02 23:18:58.204914 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:58.204889 5727 scope.go:122] "RemoveContainer" containerID="0468ea4cc006b47d022cb161a3f61aa38e7383dcea8e3b95f5d909809032662e" Nov 02 23:18:58.205334 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:18:58.205314 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0468ea4cc006b47d022cb161a3f61aa38e7383dcea8e3b95f5d909809032662e\": container with ID starting with 0468ea4cc006b47d022cb161a3f61aa38e7383dcea8e3b95f5d909809032662e not found: ID does not exist" containerID="0468ea4cc006b47d022cb161a3f61aa38e7383dcea8e3b95f5d909809032662e" Nov 02 23:18:58.205386 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:58.205345 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0468ea4cc006b47d022cb161a3f61aa38e7383dcea8e3b95f5d909809032662e"} err="failed to get container status \"0468ea4cc006b47d022cb161a3f61aa38e7383dcea8e3b95f5d909809032662e\": rpc error: code = NotFound desc = could not find container \"0468ea4cc006b47d022cb161a3f61aa38e7383dcea8e3b95f5d909809032662e\": container with ID starting with 0468ea4cc006b47d022cb161a3f61aa38e7383dcea8e3b95f5d909809032662e not found: ID does not exist" Nov 02 23:18:58.205386 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:58.205368 5727 scope.go:122] "RemoveContainer" containerID="8277f05dbbcf33945e862601f2786d15c7f9922dc52f06cc6be202f715f241b2" Nov 02 23:18:58.215921 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:58.215894 5727 scope.go:122] "RemoveContainer" containerID="8277f05dbbcf33945e862601f2786d15c7f9922dc52f06cc6be202f715f241b2" Nov 02 23:18:58.216213 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:18:58.216191 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8277f05dbbcf33945e862601f2786d15c7f9922dc52f06cc6be202f715f241b2\": container with ID starting with 8277f05dbbcf33945e862601f2786d15c7f9922dc52f06cc6be202f715f241b2 not found: ID does not exist" containerID="8277f05dbbcf33945e862601f2786d15c7f9922dc52f06cc6be202f715f241b2" Nov 02 23:18:58.216278 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:58.216223 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8277f05dbbcf33945e862601f2786d15c7f9922dc52f06cc6be202f715f241b2"} err="failed to get container status \"8277f05dbbcf33945e862601f2786d15c7f9922dc52f06cc6be202f715f241b2\": rpc error: code = NotFound desc = could not find container \"8277f05dbbcf33945e862601f2786d15c7f9922dc52f06cc6be202f715f241b2\": container with ID starting with 8277f05dbbcf33945e862601f2786d15c7f9922dc52f06cc6be202f715f241b2 not found: ID does not exist" Nov 02 23:18:58.353038 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:58.352998 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bf51d6-6db79f996c-6c4dz"] Nov 02 23:18:58.358748 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:58.358717 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bf51d6-6db79f996c-6c4dz"] Nov 02 23:18:58.465587 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:58.465498 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bf51d6-8f6bfb495-bqqqx"] Nov 02 23:18:58.472439 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:58.472408 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bf51d6-8f6bfb495-bqqqx"] Nov 02 23:18:59.916156 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:59.915530 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f7317cd-3f9f-4e50-a016-d18de7b5e8a6" path="/var/lib/kubelet/pods/6f7317cd-3f9f-4e50-a016-d18de7b5e8a6/volumes" Nov 02 23:18:59.916674 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:18:59.916261 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0d6dc88-070f-4891-8e58-8adb3606cde0" path="/var/lib/kubelet/pods/d0d6dc88-070f-4891-8e58-8adb3606cde0/volumes" Nov 02 23:19:12.548602 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:12.548553 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bf51d6-666868699c-h548g"] Nov 02 23:19:12.549144 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:12.548835 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf51d6-666868699c-h548g" podUID="96e0cb73-16c1-4cb9-9e37-8276276f4b44" containerName="maas-api" containerID="cri-o://46ecd45e2bbda2c52ae4fe8caacf70487116714e73d1f9b792604725fb46efc2" gracePeriod=30 Nov 02 23:19:12.944927 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:12.944566 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf51d6-666868699c-h548g" Nov 02 23:19:13.005237 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:13.005202 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96e0cb73-16c1-4cb9-9e37-8276276f4b44-kube-api-access-c4hw2\" (UniqueName: \"kubernetes.io/projected/96e0cb73-16c1-4cb9-9e37-8276276f4b44-kube-api-access-c4hw2\") pod \"96e0cb73-16c1-4cb9-9e37-8276276f4b44\" (UID: \"96e0cb73-16c1-4cb9-9e37-8276276f4b44\") " Nov 02 23:19:13.005422 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:13.005302 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96e0cb73-16c1-4cb9-9e37-8276276f4b44-metrics-certs\" (UniqueName: \"kubernetes.io/secret/96e0cb73-16c1-4cb9-9e37-8276276f4b44-metrics-certs\") pod \"96e0cb73-16c1-4cb9-9e37-8276276f4b44\" (UID: \"96e0cb73-16c1-4cb9-9e37-8276276f4b44\") " Nov 02 23:19:13.005422 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:13.005339 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96e0cb73-16c1-4cb9-9e37-8276276f4b44-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96e0cb73-16c1-4cb9-9e37-8276276f4b44-openshift-service-ca\") pod \"96e0cb73-16c1-4cb9-9e37-8276276f4b44\" (UID: \"96e0cb73-16c1-4cb9-9e37-8276276f4b44\") " Nov 02 23:19:13.005422 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:13.005401 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96e0cb73-16c1-4cb9-9e37-8276276f4b44-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96e0cb73-16c1-4cb9-9e37-8276276f4b44-maas-api-tls\") pod \"96e0cb73-16c1-4cb9-9e37-8276276f4b44\" (UID: \"96e0cb73-16c1-4cb9-9e37-8276276f4b44\") " Nov 02 23:19:13.006121 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:13.006012 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96e0cb73-16c1-4cb9-9e37-8276276f4b44-openshift-service-ca" pod "96e0cb73-16c1-4cb9-9e37-8276276f4b44" (UID: "96e0cb73-16c1-4cb9-9e37-8276276f4b44"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:19:13.007862 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:13.007832 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e0cb73-16c1-4cb9-9e37-8276276f4b44-kube-api-access-c4hw2" pod "96e0cb73-16c1-4cb9-9e37-8276276f4b44" (UID: "96e0cb73-16c1-4cb9-9e37-8276276f4b44"). InnerVolumeSpecName "kube-api-access-c4hw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:13.008514 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:13.008349 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96e0cb73-16c1-4cb9-9e37-8276276f4b44-maas-api-tls" pod "96e0cb73-16c1-4cb9-9e37-8276276f4b44" (UID: "96e0cb73-16c1-4cb9-9e37-8276276f4b44"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:19:13.009119 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:13.009083 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96e0cb73-16c1-4cb9-9e37-8276276f4b44-metrics-certs" pod "96e0cb73-16c1-4cb9-9e37-8276276f4b44" (UID: "96e0cb73-16c1-4cb9-9e37-8276276f4b44"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:19:13.106791 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:13.106753 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96e0cb73-16c1-4cb9-9e37-8276276f4b44-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:13.106791 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:13.106788 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96e0cb73-16c1-4cb9-9e37-8276276f4b44-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:13.106791 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:13.106798 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96e0cb73-16c1-4cb9-9e37-8276276f4b44-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:13.107112 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:13.106807 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4hw2\" (UniqueName: \"kubernetes.io/projected/96e0cb73-16c1-4cb9-9e37-8276276f4b44-kube-api-access-c4hw2\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:13.326649 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:13.326613 5727 generic.go:356] "Generic (PLEG): container finished" podID="96e0cb73-16c1-4cb9-9e37-8276276f4b44" containerID="46ecd45e2bbda2c52ae4fe8caacf70487116714e73d1f9b792604725fb46efc2" exitCode=0 Nov 02 23:19:13.326851 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:13.326663 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf51d6-666868699c-h548g" Nov 02 23:19:13.326851 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:13.326694 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf51d6-666868699c-h548g" event={"ID":"96e0cb73-16c1-4cb9-9e37-8276276f4b44","Type":"ContainerDied","Data":"46ecd45e2bbda2c52ae4fe8caacf70487116714e73d1f9b792604725fb46efc2"} Nov 02 23:19:13.326851 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:13.326751 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf51d6-666868699c-h548g" event={"ID":"96e0cb73-16c1-4cb9-9e37-8276276f4b44","Type":"ContainerDied","Data":"6a34f4c2139e5f5a5c5aa89f301be8763c753850c5ba5e8afccfaf84735ef962"} Nov 02 23:19:13.326851 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:13.326774 5727 scope.go:122] "RemoveContainer" containerID="46ecd45e2bbda2c52ae4fe8caacf70487116714e73d1f9b792604725fb46efc2" Nov 02 23:19:13.341341 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:13.341312 5727 scope.go:122] "RemoveContainer" containerID="46ecd45e2bbda2c52ae4fe8caacf70487116714e73d1f9b792604725fb46efc2" Nov 02 23:19:13.341698 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:19:13.341672 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46ecd45e2bbda2c52ae4fe8caacf70487116714e73d1f9b792604725fb46efc2\": container with ID starting with 46ecd45e2bbda2c52ae4fe8caacf70487116714e73d1f9b792604725fb46efc2 not found: ID does not exist" containerID="46ecd45e2bbda2c52ae4fe8caacf70487116714e73d1f9b792604725fb46efc2" Nov 02 23:19:13.341779 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:13.341712 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46ecd45e2bbda2c52ae4fe8caacf70487116714e73d1f9b792604725fb46efc2"} err="failed to get container status \"46ecd45e2bbda2c52ae4fe8caacf70487116714e73d1f9b792604725fb46efc2\": rpc error: code = NotFound desc = could not find container \"46ecd45e2bbda2c52ae4fe8caacf70487116714e73d1f9b792604725fb46efc2\": container with ID starting with 46ecd45e2bbda2c52ae4fe8caacf70487116714e73d1f9b792604725fb46efc2 not found: ID does not exist" Nov 02 23:19:13.492592 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:13.492482 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bf51d6-666868699c-h548g"] Nov 02 23:19:13.497110 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:13.497073 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bf51d6-666868699c-h548g"] Nov 02 23:19:13.885368 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:13.885333 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96e0cb73-16c1-4cb9-9e37-8276276f4b44" path="/var/lib/kubelet/pods/96e0cb73-16c1-4cb9-9e37-8276276f4b44/volumes" Nov 02 23:19:27.847971 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:19:27.847933 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:19:37.320073 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:37.320034 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl"] Nov 02 23:19:37.320609 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:37.320268 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl" podUID="d1a789e5-5998-4ad8-9198-7dee53101d6d" containerName="maas-api" containerID="cri-o://a12ba56632bb57ee040504633be8d517c095466d167c4bfd405f83e8c5f8303d" gracePeriod=30 Nov 02 23:19:37.592112 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:37.592026 5727 generic.go:356] "Generic (PLEG): container finished" podID="d1a789e5-5998-4ad8-9198-7dee53101d6d" containerID="a12ba56632bb57ee040504633be8d517c095466d167c4bfd405f83e8c5f8303d" exitCode=0 Nov 02 23:19:37.592364 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:37.592332 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl" event={"ID":"d1a789e5-5998-4ad8-9198-7dee53101d6d","Type":"ContainerDied","Data":"a12ba56632bb57ee040504633be8d517c095466d167c4bfd405f83e8c5f8303d"} Nov 02 23:19:37.639533 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:37.639502 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl" Nov 02 23:19:37.736001 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:37.735967 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1a789e5-5998-4ad8-9198-7dee53101d6d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1a789e5-5998-4ad8-9198-7dee53101d6d-maas-api-tls\") pod \"d1a789e5-5998-4ad8-9198-7dee53101d6d\" (UID: \"d1a789e5-5998-4ad8-9198-7dee53101d6d\") " Nov 02 23:19:37.736191 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:37.736052 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1a789e5-5998-4ad8-9198-7dee53101d6d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1a789e5-5998-4ad8-9198-7dee53101d6d-openshift-service-ca\") pod \"d1a789e5-5998-4ad8-9198-7dee53101d6d\" (UID: \"d1a789e5-5998-4ad8-9198-7dee53101d6d\") " Nov 02 23:19:37.736246 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:37.736195 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1a789e5-5998-4ad8-9198-7dee53101d6d-kube-api-access-rh6jg\" (UniqueName: \"kubernetes.io/projected/d1a789e5-5998-4ad8-9198-7dee53101d6d-kube-api-access-rh6jg\") pod \"d1a789e5-5998-4ad8-9198-7dee53101d6d\" (UID: \"d1a789e5-5998-4ad8-9198-7dee53101d6d\") " Nov 02 23:19:37.736328 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:37.736308 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1a789e5-5998-4ad8-9198-7dee53101d6d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1a789e5-5998-4ad8-9198-7dee53101d6d-metrics-certs\") pod \"d1a789e5-5998-4ad8-9198-7dee53101d6d\" (UID: \"d1a789e5-5998-4ad8-9198-7dee53101d6d\") " Nov 02 23:19:37.736479 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:37.736452 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1a789e5-5998-4ad8-9198-7dee53101d6d-openshift-service-ca" pod "d1a789e5-5998-4ad8-9198-7dee53101d6d" (UID: "d1a789e5-5998-4ad8-9198-7dee53101d6d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:19:37.736890 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:37.736866 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1a789e5-5998-4ad8-9198-7dee53101d6d-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:37.738667 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:37.738637 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a789e5-5998-4ad8-9198-7dee53101d6d-kube-api-access-rh6jg" pod "d1a789e5-5998-4ad8-9198-7dee53101d6d" (UID: "d1a789e5-5998-4ad8-9198-7dee53101d6d"). InnerVolumeSpecName "kube-api-access-rh6jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:37.738929 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:37.738907 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1a789e5-5998-4ad8-9198-7dee53101d6d-maas-api-tls" pod "d1a789e5-5998-4ad8-9198-7dee53101d6d" (UID: "d1a789e5-5998-4ad8-9198-7dee53101d6d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:19:37.738994 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:37.738926 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1a789e5-5998-4ad8-9198-7dee53101d6d-metrics-certs" pod "d1a789e5-5998-4ad8-9198-7dee53101d6d" (UID: "d1a789e5-5998-4ad8-9198-7dee53101d6d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:19:37.839001 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:37.838948 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1a789e5-5998-4ad8-9198-7dee53101d6d-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:37.839001 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:37.838992 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1a789e5-5998-4ad8-9198-7dee53101d6d-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:37.839001 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:37.839007 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh6jg\" (UniqueName: \"kubernetes.io/projected/d1a789e5-5998-4ad8-9198-7dee53101d6d-kube-api-access-rh6jg\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:38.609486 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:38.609453 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl" Nov 02 23:19:38.609486 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:38.609474 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl" event={"ID":"d1a789e5-5998-4ad8-9198-7dee53101d6d","Type":"ContainerDied","Data":"c00368316c6b25786db8d0b06ea4ee393f323d368d1219d975d2852926208d4b"} Nov 02 23:19:38.610013 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:38.609544 5727 scope.go:122] "RemoveContainer" containerID="a12ba56632bb57ee040504633be8d517c095466d167c4bfd405f83e8c5f8303d" Nov 02 23:19:38.774503 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:38.774468 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl"] Nov 02 23:19:38.823878 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:38.823845 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-048ef3-5b88d9c9cd-jmjtl"] Nov 02 23:19:39.884355 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:39.884310 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1a789e5-5998-4ad8-9198-7dee53101d6d" path="/var/lib/kubelet/pods/d1a789e5-5998-4ad8-9198-7dee53101d6d/volumes" Nov 02 23:19:54.621705 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:54.621670 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml"] Nov 02 23:19:54.622171 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:54.621913 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" podUID="6f88cb2c-1bec-4b58-a525-326c52c23931" containerName="istio-proxy" containerID="cri-o://cac15f0d77fd386685429e35235a9e9fbf397d066b02006a42c30b02cb552102" gracePeriod=30 Nov 02 23:19:56.622706 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:56.622664 5727 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:19:56.623229 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:56.622748 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" podUID="6f88cb2c-1bec-4b58-a525-326c52c23931" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:19:57.846743 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:19:57.846706 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:19:59.777305 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:59.777264 5727 generic.go:356] "Generic (PLEG): container finished" podID="6f88cb2c-1bec-4b58-a525-326c52c23931" containerID="cac15f0d77fd386685429e35235a9e9fbf397d066b02006a42c30b02cb552102" exitCode=0 Nov 02 23:19:59.777724 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:59.777547 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" event={"ID":"6f88cb2c-1bec-4b58-a525-326c52c23931","Type":"ContainerDied","Data":"cac15f0d77fd386685429e35235a9e9fbf397d066b02006a42c30b02cb552102"} Nov 02 23:19:59.907995 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:59.907933 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:19:59.972092 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:59.972057 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f88cb2c-1bec-4b58-a525-326c52c23931-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f88cb2c-1bec-4b58-a525-326c52c23931-istio-podinfo\") pod \"6f88cb2c-1bec-4b58-a525-326c52c23931\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " Nov 02 23:19:59.972282 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:59.972262 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-workload-socket\") pod \"6f88cb2c-1bec-4b58-a525-326c52c23931\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " Nov 02 23:19:59.972384 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:59.972361 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f88cb2c-1bec-4b58-a525-326c52c23931-istio-token\" (UniqueName: \"kubernetes.io/projected/6f88cb2c-1bec-4b58-a525-326c52c23931-istio-token\") pod \"6f88cb2c-1bec-4b58-a525-326c52c23931\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " Nov 02 23:19:59.972554 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:59.972415 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f88cb2c-1bec-4b58-a525-326c52c23931-kube-api-access-zj8fq\" (UniqueName: \"kubernetes.io/projected/6f88cb2c-1bec-4b58-a525-326c52c23931-kube-api-access-zj8fq\") pod \"6f88cb2c-1bec-4b58-a525-326c52c23931\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " Nov 02 23:19:59.972554 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:59.972444 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-istio-data\") pod \"6f88cb2c-1bec-4b58-a525-326c52c23931\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " Nov 02 23:19:59.972554 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:59.972507 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-workload-certs\") pod \"6f88cb2c-1bec-4b58-a525-326c52c23931\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " Nov 02 23:19:59.972554 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:59.972540 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-istio-envoy\") pod \"6f88cb2c-1bec-4b58-a525-326c52c23931\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " Nov 02 23:19:59.972782 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:59.972575 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-workload-socket" pod "6f88cb2c-1bec-4b58-a525-326c52c23931" (UID: "6f88cb2c-1bec-4b58-a525-326c52c23931"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:59.972782 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:59.972586 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f88cb2c-1bec-4b58-a525-326c52c23931-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f88cb2c-1bec-4b58-a525-326c52c23931-istiod-ca-cert\") pod \"6f88cb2c-1bec-4b58-a525-326c52c23931\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " Nov 02 23:19:59.972782 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:59.972706 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-credential-socket\") pod \"6f88cb2c-1bec-4b58-a525-326c52c23931\" (UID: \"6f88cb2c-1bec-4b58-a525-326c52c23931\") " Nov 02 23:19:59.973111 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:59.972936 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f88cb2c-1bec-4b58-a525-326c52c23931-istiod-ca-cert" pod "6f88cb2c-1bec-4b58-a525-326c52c23931" (UID: "6f88cb2c-1bec-4b58-a525-326c52c23931"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:19:59.973111 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:59.972943 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-workload-certs" pod "6f88cb2c-1bec-4b58-a525-326c52c23931" (UID: "6f88cb2c-1bec-4b58-a525-326c52c23931"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:59.973111 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:59.973058 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-istio-data" pod "6f88cb2c-1bec-4b58-a525-326c52c23931" (UID: "6f88cb2c-1bec-4b58-a525-326c52c23931"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:59.973348 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:59.973315 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-credential-socket" pod "6f88cb2c-1bec-4b58-a525-326c52c23931" (UID: "6f88cb2c-1bec-4b58-a525-326c52c23931"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:59.973785 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:59.973690 5727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-istio-data\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:59.973785 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:59.973717 5727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-workload-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:59.973785 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:59.973732 5727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f88cb2c-1bec-4b58-a525-326c52c23931-istiod-ca-cert\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:59.973785 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:59.973746 5727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-credential-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:59.973785 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:59.973760 5727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-workload-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:59.974666 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:59.974643 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f88cb2c-1bec-4b58-a525-326c52c23931-istio-podinfo" pod "6f88cb2c-1bec-4b58-a525-326c52c23931" (UID: "6f88cb2c-1bec-4b58-a525-326c52c23931"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:19:59.975279 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:59.975252 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f88cb2c-1bec-4b58-a525-326c52c23931-istio-token" pod "6f88cb2c-1bec-4b58-a525-326c52c23931" (UID: "6f88cb2c-1bec-4b58-a525-326c52c23931"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:59.975976 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:59.975946 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f88cb2c-1bec-4b58-a525-326c52c23931-kube-api-access-zj8fq" pod "6f88cb2c-1bec-4b58-a525-326c52c23931" (UID: "6f88cb2c-1bec-4b58-a525-326c52c23931"). InnerVolumeSpecName "kube-api-access-zj8fq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:59.976064 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:19:59.976041 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-istio-envoy" pod "6f88cb2c-1bec-4b58-a525-326c52c23931" (UID: "6f88cb2c-1bec-4b58-a525-326c52c23931"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:00.075629 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:00.075590 5727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f88cb2c-1bec-4b58-a525-326c52c23931-istio-token\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:00.075629 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:00.075632 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj8fq\" (UniqueName: \"kubernetes.io/projected/6f88cb2c-1bec-4b58-a525-326c52c23931-kube-api-access-zj8fq\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:00.075859 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:00.075650 5727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f88cb2c-1bec-4b58-a525-326c52c23931-istio-envoy\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:00.075859 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:00.075664 5727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f88cb2c-1bec-4b58-a525-326c52c23931-istio-podinfo\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:00.290024 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:00.289942 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv"] Nov 02 23:20:00.290264 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:00.290235 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv" podUID="c2e0f11e-4b44-4088-b095-7c9e47a07707" containerName="maas-api" containerID="cri-o://961888e46073aaf0ade78eae180b6a5d67483fb2ea55a31bc6f6c03998c8180b" gracePeriod=30 Nov 02 23:20:00.582802 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:00.582771 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv" Nov 02 23:20:00.689608 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:00.689571 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2e0f11e-4b44-4088-b095-7c9e47a07707-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2e0f11e-4b44-4088-b095-7c9e47a07707-metrics-certs\") pod \"c2e0f11e-4b44-4088-b095-7c9e47a07707\" (UID: \"c2e0f11e-4b44-4088-b095-7c9e47a07707\") " Nov 02 23:20:00.689900 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:00.689703 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2e0f11e-4b44-4088-b095-7c9e47a07707-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e0f11e-4b44-4088-b095-7c9e47a07707-maas-api-tls\") pod \"c2e0f11e-4b44-4088-b095-7c9e47a07707\" (UID: \"c2e0f11e-4b44-4088-b095-7c9e47a07707\") " Nov 02 23:20:00.689900 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:00.689800 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2e0f11e-4b44-4088-b095-7c9e47a07707-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e0f11e-4b44-4088-b095-7c9e47a07707-openshift-service-ca\") pod \"c2e0f11e-4b44-4088-b095-7c9e47a07707\" (UID: \"c2e0f11e-4b44-4088-b095-7c9e47a07707\") " Nov 02 23:20:00.689900 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:00.689835 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2e0f11e-4b44-4088-b095-7c9e47a07707-kube-api-access-skfvf\" (UniqueName: \"kubernetes.io/projected/c2e0f11e-4b44-4088-b095-7c9e47a07707-kube-api-access-skfvf\") pod \"c2e0f11e-4b44-4088-b095-7c9e47a07707\" (UID: \"c2e0f11e-4b44-4088-b095-7c9e47a07707\") " Nov 02 23:20:00.691542 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:00.691481 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2e0f11e-4b44-4088-b095-7c9e47a07707-openshift-service-ca" pod "c2e0f11e-4b44-4088-b095-7c9e47a07707" (UID: "c2e0f11e-4b44-4088-b095-7c9e47a07707"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:20:00.693495 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:00.693463 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2e0f11e-4b44-4088-b095-7c9e47a07707-metrics-certs" pod "c2e0f11e-4b44-4088-b095-7c9e47a07707" (UID: "c2e0f11e-4b44-4088-b095-7c9e47a07707"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:20:00.693794 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:00.693766 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2e0f11e-4b44-4088-b095-7c9e47a07707-kube-api-access-skfvf" pod "c2e0f11e-4b44-4088-b095-7c9e47a07707" (UID: "c2e0f11e-4b44-4088-b095-7c9e47a07707"). InnerVolumeSpecName "kube-api-access-skfvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:20:00.693974 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:00.693956 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2e0f11e-4b44-4088-b095-7c9e47a07707-maas-api-tls" pod "c2e0f11e-4b44-4088-b095-7c9e47a07707" (UID: "c2e0f11e-4b44-4088-b095-7c9e47a07707"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:20:00.796153 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:00.791927 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e0f11e-4b44-4088-b095-7c9e47a07707-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:00.796153 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:00.791964 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e0f11e-4b44-4088-b095-7c9e47a07707-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:00.796153 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:00.791978 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skfvf\" (UniqueName: \"kubernetes.io/projected/c2e0f11e-4b44-4088-b095-7c9e47a07707-kube-api-access-skfvf\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:00.796153 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:00.791993 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2e0f11e-4b44-4088-b095-7c9e47a07707-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:00.796153 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:00.795239 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" Nov 02 23:20:00.796153 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:00.795376 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml" event={"ID":"6f88cb2c-1bec-4b58-a525-326c52c23931","Type":"ContainerDied","Data":"47ebfef07de5c1bae3c5808ac6ab06f231498bb00eb62af593c88c33aad013cf"} Nov 02 23:20:00.796153 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:00.795425 5727 scope.go:122] "RemoveContainer" containerID="cac15f0d77fd386685429e35235a9e9fbf397d066b02006a42c30b02cb552102" Nov 02 23:20:00.804659 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:00.799354 5727 generic.go:356] "Generic (PLEG): container finished" podID="c2e0f11e-4b44-4088-b095-7c9e47a07707" containerID="961888e46073aaf0ade78eae180b6a5d67483fb2ea55a31bc6f6c03998c8180b" exitCode=0 Nov 02 23:20:00.804659 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:00.799521 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv" event={"ID":"c2e0f11e-4b44-4088-b095-7c9e47a07707","Type":"ContainerDied","Data":"961888e46073aaf0ade78eae180b6a5d67483fb2ea55a31bc6f6c03998c8180b"} Nov 02 23:20:00.804659 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:00.799592 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv" event={"ID":"c2e0f11e-4b44-4088-b095-7c9e47a07707","Type":"ContainerDied","Data":"cb8bfb79dda3b7aaa6cb6a34cd7f5256c3dd51405f9785a492bac655307d9b7c"} Nov 02 23:20:00.804659 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:00.799687 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv" Nov 02 23:20:00.831839 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:00.831789 5727 scope.go:122] "RemoveContainer" containerID="961888e46073aaf0ade78eae180b6a5d67483fb2ea55a31bc6f6c03998c8180b" Nov 02 23:20:00.860213 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:00.860184 5727 scope.go:122] "RemoveContainer" containerID="961888e46073aaf0ade78eae180b6a5d67483fb2ea55a31bc6f6c03998c8180b" Nov 02 23:20:00.860600 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:20:00.860574 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"961888e46073aaf0ade78eae180b6a5d67483fb2ea55a31bc6f6c03998c8180b\": container with ID starting with 961888e46073aaf0ade78eae180b6a5d67483fb2ea55a31bc6f6c03998c8180b not found: ID does not exist" containerID="961888e46073aaf0ade78eae180b6a5d67483fb2ea55a31bc6f6c03998c8180b" Nov 02 23:20:00.860675 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:00.860613 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"961888e46073aaf0ade78eae180b6a5d67483fb2ea55a31bc6f6c03998c8180b"} err="failed to get container status \"961888e46073aaf0ade78eae180b6a5d67483fb2ea55a31bc6f6c03998c8180b\": rpc error: code = NotFound desc = could not find container \"961888e46073aaf0ade78eae180b6a5d67483fb2ea55a31bc6f6c03998c8180b\": container with ID starting with 961888e46073aaf0ade78eae180b6a5d67483fb2ea55a31bc6f6c03998c8180b not found: ID does not exist" Nov 02 23:20:00.967245 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:00.967208 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml"] Nov 02 23:20:00.968315 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:00.968290 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-7qkml"] Nov 02 23:20:01.114106 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:01.114013 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv"] Nov 02 23:20:01.122148 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:01.121386 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a7a67875-5ccb7d57c4-g5pnv"] Nov 02 23:20:01.884813 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:01.884772 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f88cb2c-1bec-4b58-a525-326c52c23931" path="/var/lib/kubelet/pods/6f88cb2c-1bec-4b58-a525-326c52c23931/volumes" Nov 02 23:20:01.885455 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:01.885437 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2e0f11e-4b44-4088-b095-7c9e47a07707" path="/var/lib/kubelet/pods/c2e0f11e-4b44-4088-b095-7c9e47a07707/volumes" Nov 02 23:20:27.862779 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:20:27.862738 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:20:36.121624 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.121591 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-6844b65d7-d9v9r"] Nov 02 23:20:36.122218 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.122204 5727 cpu_manager.go:414] "containerMap: removing container" podUID="d1a789e5-5998-4ad8-9198-7dee53101d6d" containerName="maas-api" Nov 02 23:20:36.122276 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.122218 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1a789e5-5998-4ad8-9198-7dee53101d6d" containerName="maas-api" Nov 02 23:20:36.122276 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.122236 5727 cpu_manager.go:414] "containerMap: removing container" podUID="6f88cb2c-1bec-4b58-a525-326c52c23931" containerName="istio-proxy" Nov 02 23:20:36.122276 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.122241 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f88cb2c-1bec-4b58-a525-326c52c23931" containerName="istio-proxy" Nov 02 23:20:36.122276 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.122248 5727 cpu_manager.go:414] "containerMap: removing container" podUID="96e0cb73-16c1-4cb9-9e37-8276276f4b44" containerName="maas-api" Nov 02 23:20:36.122276 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.122254 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96e0cb73-16c1-4cb9-9e37-8276276f4b44" containerName="maas-api" Nov 02 23:20:36.122276 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.122266 5727 cpu_manager.go:414] "containerMap: removing container" podUID="d0d6dc88-070f-4891-8e58-8adb3606cde0" containerName="payload-pre-processing" Nov 02 23:20:36.122276 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.122271 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0d6dc88-070f-4891-8e58-8adb3606cde0" containerName="payload-pre-processing" Nov 02 23:20:36.122489 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.122289 5727 cpu_manager.go:414] "containerMap: removing container" podUID="11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d" containerName="maas-api" Nov 02 23:20:36.122489 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.122294 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d" containerName="maas-api" Nov 02 23:20:36.122489 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.122312 5727 cpu_manager.go:414] "containerMap: removing container" podUID="6f7317cd-3f9f-4e50-a016-d18de7b5e8a6" containerName="payload-processing" Nov 02 23:20:36.122489 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.122317 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f7317cd-3f9f-4e50-a016-d18de7b5e8a6" containerName="payload-processing" Nov 02 23:20:36.122489 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.122330 5727 cpu_manager.go:414] "containerMap: removing container" podUID="c2e0f11e-4b44-4088-b095-7c9e47a07707" containerName="maas-api" Nov 02 23:20:36.122489 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.122336 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2e0f11e-4b44-4088-b095-7c9e47a07707" containerName="maas-api" Nov 02 23:20:36.122489 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.122426 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1a789e5-5998-4ad8-9198-7dee53101d6d" containerName="maas-api" Nov 02 23:20:36.122489 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.122436 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f88cb2c-1bec-4b58-a525-326c52c23931" containerName="istio-proxy" Nov 02 23:20:36.122489 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.122441 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="96e0cb73-16c1-4cb9-9e37-8276276f4b44" containerName="maas-api" Nov 02 23:20:36.122489 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.122448 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0d6dc88-070f-4891-8e58-8adb3606cde0" containerName="payload-pre-processing" Nov 02 23:20:36.122489 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.122457 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="11fb6a0c-cf3a-4dbe-b1b4-a727c5b4c81d" containerName="maas-api" Nov 02 23:20:36.122489 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.122464 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f7317cd-3f9f-4e50-a016-d18de7b5e8a6" containerName="payload-processing" Nov 02 23:20:36.122489 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.122470 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2e0f11e-4b44-4088-b095-7c9e47a07707" containerName="maas-api" Nov 02 23:20:36.127529 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.127501 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-6844b65d7-d9v9r" Nov 02 23:20:36.183098 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.183063 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-6844b65d7-d9v9r"] Nov 02 23:20:36.218305 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.218209 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw7bh\" (UniqueName: \"kubernetes.io/projected/c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa-kube-api-access-qw7bh\") pod \"payload-pre-processing-e2e-ipp-b-92072f-6844b65d7-d9v9r\" (UID: \"c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-6844b65d7-d9v9r" Nov 02 23:20:36.218890 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.218858 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-92072f-6844b65d7-d9v9r\" (UID: \"c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-6844b65d7-d9v9r" Nov 02 23:20:36.231452 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.231408 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-92072f-dockercfg-2tjxs\"" Nov 02 23:20:36.231651 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.231542 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-92072f\"" Nov 02 23:20:36.279874 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.278171 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-92072f-5bf4fb99-kn2nd"] Nov 02 23:20:36.289500 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.289467 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-5bf4fb99-kn2nd" Nov 02 23:20:36.320657 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.320620 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ac2e404-7f02-4318-9aaf-0c33d1e99124-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-92072f-5bf4fb99-kn2nd\" (UID: \"3ac2e404-7f02-4318-9aaf-0c33d1e99124\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-5bf4fb99-kn2nd" Nov 02 23:20:36.321051 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.321030 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw7bh\" (UniqueName: \"kubernetes.io/projected/c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa-kube-api-access-qw7bh\") pod \"payload-pre-processing-e2e-ipp-b-92072f-6844b65d7-d9v9r\" (UID: \"c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-6844b65d7-d9v9r" Nov 02 23:20:36.321931 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.321907 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-92072f-6844b65d7-d9v9r\" (UID: \"c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-6844b65d7-d9v9r" Nov 02 23:20:36.323011 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.322985 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx8qf\" (UniqueName: \"kubernetes.io/projected/3ac2e404-7f02-4318-9aaf-0c33d1e99124-kube-api-access-xx8qf\") pod \"payload-processing-e2e-ipp-b-92072f-5bf4fb99-kn2nd\" (UID: \"3ac2e404-7f02-4318-9aaf-0c33d1e99124\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-5bf4fb99-kn2nd" Nov 02 23:20:36.323246 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.322848 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-92072f-6844b65d7-d9v9r\" (UID: \"c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-6844b65d7-d9v9r" Nov 02 23:20:36.335371 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.335341 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-92072f-5bf4fb99-kn2nd"] Nov 02 23:20:36.382006 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.381941 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw7bh\" (UniqueName: \"kubernetes.io/projected/c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa-kube-api-access-qw7bh\") pod \"payload-pre-processing-e2e-ipp-b-92072f-6844b65d7-d9v9r\" (UID: \"c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-6844b65d7-d9v9r" Nov 02 23:20:36.424850 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.424813 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ac2e404-7f02-4318-9aaf-0c33d1e99124-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-92072f-5bf4fb99-kn2nd\" (UID: \"3ac2e404-7f02-4318-9aaf-0c33d1e99124\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-5bf4fb99-kn2nd" Nov 02 23:20:36.425059 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.424975 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx8qf\" (UniqueName: \"kubernetes.io/projected/3ac2e404-7f02-4318-9aaf-0c33d1e99124-kube-api-access-xx8qf\") pod \"payload-processing-e2e-ipp-b-92072f-5bf4fb99-kn2nd\" (UID: \"3ac2e404-7f02-4318-9aaf-0c33d1e99124\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-5bf4fb99-kn2nd" Nov 02 23:20:36.425665 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.425642 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ac2e404-7f02-4318-9aaf-0c33d1e99124-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-92072f-5bf4fb99-kn2nd\" (UID: \"3ac2e404-7f02-4318-9aaf-0c33d1e99124\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-5bf4fb99-kn2nd" Nov 02 23:20:36.449005 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.448976 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-6844b65d7-d9v9r" Nov 02 23:20:36.485860 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.485830 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx8qf\" (UniqueName: \"kubernetes.io/projected/3ac2e404-7f02-4318-9aaf-0c33d1e99124-kube-api-access-xx8qf\") pod \"payload-processing-e2e-ipp-b-92072f-5bf4fb99-kn2nd\" (UID: \"3ac2e404-7f02-4318-9aaf-0c33d1e99124\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-5bf4fb99-kn2nd" Nov 02 23:20:36.622281 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.622246 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-5bf4fb99-kn2nd" Nov 02 23:20:36.833544 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:36.833512 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-6844b65d7-d9v9r"] Nov 02 23:20:36.833650 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:20:36.833597 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc703dc6f_7178_43b9_b5fa_9de1e1b9c2fa.slice/crio-d9eccd301426fc5238ff29cf3dd2650337834aa972a9b6c87dbfb80908cb3afc WatchSource:0}: Error finding container d9eccd301426fc5238ff29cf3dd2650337834aa972a9b6c87dbfb80908cb3afc: Status 404 returned error can't find the container with id d9eccd301426fc5238ff29cf3dd2650337834aa972a9b6c87dbfb80908cb3afc Nov 02 23:20:37.002229 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:37.002199 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-92072f-5bf4fb99-kn2nd"] Nov 02 23:20:37.002752 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:20:37.002722 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac2e404_7f02_4318_9aaf_0c33d1e99124.slice/crio-397afb4656ca8d6b04c82bf58019971db8e48afcdcd39cf485c4c73a189c7fd3 WatchSource:0}: Error finding container 397afb4656ca8d6b04c82bf58019971db8e48afcdcd39cf485c4c73a189c7fd3: Status 404 returned error can't find the container with id 397afb4656ca8d6b04c82bf58019971db8e48afcdcd39cf485c4c73a189c7fd3 Nov 02 23:20:37.129549 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:37.129505 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-6844b65d7-d9v9r" event={"ID":"c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa","Type":"ContainerStarted","Data":"71882a6d1a3ec3bdb70bc44fc49948a80d9ece6a478ce040fa031c12f6e325fe"} Nov 02 23:20:37.129937 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:37.129568 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-6844b65d7-d9v9r" event={"ID":"c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa","Type":"ContainerStarted","Data":"d9eccd301426fc5238ff29cf3dd2650337834aa972a9b6c87dbfb80908cb3afc"} Nov 02 23:20:37.129937 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:37.129730 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-6844b65d7-d9v9r" Nov 02 23:20:37.131524 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:37.131495 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-5bf4fb99-kn2nd" event={"ID":"3ac2e404-7f02-4318-9aaf-0c33d1e99124","Type":"ContainerStarted","Data":"0b687f022d87bb3de9aaf69521a5cf9e44f8472e88e15f76c6fd22365aefa79f"} Nov 02 23:20:37.131687 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:37.131545 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-5bf4fb99-kn2nd" event={"ID":"3ac2e404-7f02-4318-9aaf-0c33d1e99124","Type":"ContainerStarted","Data":"397afb4656ca8d6b04c82bf58019971db8e48afcdcd39cf485c4c73a189c7fd3"} Nov 02 23:20:37.131687 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:37.131573 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-5bf4fb99-kn2nd" Nov 02 23:20:37.245468 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:37.245346 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-6844b65d7-d9v9r" podStartSLOduration=1.245322499 podStartE2EDuration="1.245322499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:20:37.243710293 +0000 UTC m=+1923.482207982" watchObservedRunningTime="2026-11-02 23:20:37.245322499 +0000 UTC m=+1923.483820158" Nov 02 23:20:37.365803 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:37.365730 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-5bf4fb99-kn2nd" podStartSLOduration=1.365709822 podStartE2EDuration="1.365709822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:20:37.357972843 +0000 UTC m=+1923.596470529" watchObservedRunningTime="2026-11-02 23:20:37.365709822 +0000 UTC m=+1923.604207484" Nov 02 23:20:48.142693 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:48.142662 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-5bf4fb99-kn2nd" Nov 02 23:20:48.143151 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:20:48.142730 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-6844b65d7-d9v9r" Nov 02 23:20:57.847409 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:20:57.847368 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:21:27.847239 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:21:27.847198 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:21:43.484425 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:43.484351 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9"] Nov 02 23:21:43.488455 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:43.488439 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9" Nov 02 23:21:43.591373 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:43.591344 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-92072f-serving-cert\"" Nov 02 23:21:43.595294 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:43.595269 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9"] Nov 02 23:21:43.625446 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:43.625406 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96c11959-99bc-467a-a844-74fc983c126e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9\" (UID: \"96c11959-99bc-467a-a844-74fc983c126e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9" Nov 02 23:21:43.625657 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:43.625611 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96c11959-99bc-467a-a844-74fc983c126e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9\" (UID: \"96c11959-99bc-467a-a844-74fc983c126e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9" Nov 02 23:21:43.625739 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:43.625719 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96c11959-99bc-467a-a844-74fc983c126e-metrics-certs\") pod \"maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9\" (UID: \"96c11959-99bc-467a-a844-74fc983c126e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9" Nov 02 23:21:43.625872 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:43.625853 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhnql\" (UniqueName: \"kubernetes.io/projected/96c11959-99bc-467a-a844-74fc983c126e-kube-api-access-rhnql\") pod \"maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9\" (UID: \"96c11959-99bc-467a-a844-74fc983c126e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9" Nov 02 23:21:43.727355 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:43.727325 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96c11959-99bc-467a-a844-74fc983c126e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9\" (UID: \"96c11959-99bc-467a-a844-74fc983c126e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9" Nov 02 23:21:43.727544 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:43.727388 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96c11959-99bc-467a-a844-74fc983c126e-metrics-certs\") pod \"maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9\" (UID: \"96c11959-99bc-467a-a844-74fc983c126e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9" Nov 02 23:21:43.727544 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:21:43.727478 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-92072f-serving-cert: secret "maas-api-e2e-ipp-b-92072f-serving-cert" not found Nov 02 23:21:43.727544 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:43.727508 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhnql\" (UniqueName: \"kubernetes.io/projected/96c11959-99bc-467a-a844-74fc983c126e-kube-api-access-rhnql\") pod \"maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9\" (UID: \"96c11959-99bc-467a-a844-74fc983c126e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9" Nov 02 23:21:43.727544 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:21:43.727539 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96c11959-99bc-467a-a844-74fc983c126e-maas-api-tls podName:96c11959-99bc-467a-a844-74fc983c126e nodeName:}" failed. No retries permitted until 2026-11-02 23:21:44.227524216 +0000 UTC m=+1990.466021873 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/96c11959-99bc-467a-a844-74fc983c126e-maas-api-tls") pod "maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9" (UID: "96c11959-99bc-467a-a844-74fc983c126e") : secret "maas-api-e2e-ipp-b-92072f-serving-cert" not found Nov 02 23:21:43.727716 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:43.727572 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96c11959-99bc-467a-a844-74fc983c126e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9\" (UID: \"96c11959-99bc-467a-a844-74fc983c126e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9" Nov 02 23:21:43.728238 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:43.728180 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96c11959-99bc-467a-a844-74fc983c126e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9\" (UID: \"96c11959-99bc-467a-a844-74fc983c126e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9" Nov 02 23:21:43.730106 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:43.730081 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96c11959-99bc-467a-a844-74fc983c126e-metrics-certs\") pod \"maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9\" (UID: \"96c11959-99bc-467a-a844-74fc983c126e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9" Nov 02 23:21:43.791032 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:43.790943 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhnql\" (UniqueName: \"kubernetes.io/projected/96c11959-99bc-467a-a844-74fc983c126e-kube-api-access-rhnql\") pod \"maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9\" (UID: \"96c11959-99bc-467a-a844-74fc983c126e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9" Nov 02 23:21:44.235590 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:44.235556 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96c11959-99bc-467a-a844-74fc983c126e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9\" (UID: \"96c11959-99bc-467a-a844-74fc983c126e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9" Nov 02 23:21:44.239713 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:44.239680 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96c11959-99bc-467a-a844-74fc983c126e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9\" (UID: \"96c11959-99bc-467a-a844-74fc983c126e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9" Nov 02 23:21:44.401981 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:44.401946 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9" Nov 02 23:21:44.802193 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:44.802163 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9"] Nov 02 23:21:44.802789 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:21:44.802756 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96c11959_99bc_467a_a844_74fc983c126e.slice/crio-f985ae02bf1d2443fd61a533aeb7abefdf0a382fb80846e77cc55f6d700611cc WatchSource:0}: Error finding container f985ae02bf1d2443fd61a533aeb7abefdf0a382fb80846e77cc55f6d700611cc: Status 404 returned error can't find the container with id f985ae02bf1d2443fd61a533aeb7abefdf0a382fb80846e77cc55f6d700611cc Nov 02 23:21:45.698543 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:45.698501 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9" event={"ID":"96c11959-99bc-467a-a844-74fc983c126e","Type":"ContainerStarted","Data":"f985ae02bf1d2443fd61a533aeb7abefdf0a382fb80846e77cc55f6d700611cc"} Nov 02 23:21:46.710802 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:46.710763 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9" event={"ID":"96c11959-99bc-467a-a844-74fc983c126e","Type":"ContainerStarted","Data":"a087d729445696926f6289cb9af47839f9240032608112928be53f3b8e5c6596"} Nov 02 23:21:46.711341 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:46.710881 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9" Nov 02 23:21:46.827268 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:46.827212 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9" podStartSLOduration=2.9846529569999998 podStartE2EDuration="3.827198408s" totalImagesPullingTime="842.545451ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:21:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:21:46.826601269 +0000 UTC m=+1993.065098951" watchObservedRunningTime="2026-11-02 23:21:46.827198408 +0000 UTC m=+1993.065696071" Nov 02 23:21:48.461302 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:48.461268 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc"] Nov 02 23:21:48.461701 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:48.461479 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc" podUID="19c83f05-c0d9-47f8-a8ab-5cf565015621" containerName="maas-api" containerID="cri-o://824526bfec4f627dc093d6558f6dd3a03cc5849bd3d7de7482644a80c416b842" gracePeriod=30 Nov 02 23:21:48.754464 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:48.754432 5727 generic.go:356] "Generic (PLEG): container finished" podID="19c83f05-c0d9-47f8-a8ab-5cf565015621" containerID="824526bfec4f627dc093d6558f6dd3a03cc5849bd3d7de7482644a80c416b842" exitCode=0 Nov 02 23:21:48.754623 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:48.754508 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc" event={"ID":"19c83f05-c0d9-47f8-a8ab-5cf565015621","Type":"ContainerDied","Data":"824526bfec4f627dc093d6558f6dd3a03cc5849bd3d7de7482644a80c416b842"} Nov 02 23:21:48.754623 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:48.754558 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc" event={"ID":"19c83f05-c0d9-47f8-a8ab-5cf565015621","Type":"ContainerDied","Data":"0ad39fd8ec514a15d901ace24fe5ab3834fb40c00c88d070cf394518ac5ac5e0"} Nov 02 23:21:48.754623 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:48.754575 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ad39fd8ec514a15d901ace24fe5ab3834fb40c00c88d070cf394518ac5ac5e0" Nov 02 23:21:48.764414 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:48.764391 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc" Nov 02 23:21:48.833819 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:48.833788 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19c83f05-c0d9-47f8-a8ab-5cf565015621-metrics-certs\" (UniqueName: \"kubernetes.io/secret/19c83f05-c0d9-47f8-a8ab-5cf565015621-metrics-certs\") pod \"19c83f05-c0d9-47f8-a8ab-5cf565015621\" (UID: \"19c83f05-c0d9-47f8-a8ab-5cf565015621\") " Nov 02 23:21:48.834001 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:48.833895 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19c83f05-c0d9-47f8-a8ab-5cf565015621-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19c83f05-c0d9-47f8-a8ab-5cf565015621-maas-api-tls\") pod \"19c83f05-c0d9-47f8-a8ab-5cf565015621\" (UID: \"19c83f05-c0d9-47f8-a8ab-5cf565015621\") " Nov 02 23:21:48.834048 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:48.834014 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19c83f05-c0d9-47f8-a8ab-5cf565015621-kube-api-access-98dn5\" (UniqueName: \"kubernetes.io/projected/19c83f05-c0d9-47f8-a8ab-5cf565015621-kube-api-access-98dn5\") pod \"19c83f05-c0d9-47f8-a8ab-5cf565015621\" (UID: \"19c83f05-c0d9-47f8-a8ab-5cf565015621\") " Nov 02 23:21:48.834094 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:48.834079 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19c83f05-c0d9-47f8-a8ab-5cf565015621-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19c83f05-c0d9-47f8-a8ab-5cf565015621-openshift-service-ca\") pod \"19c83f05-c0d9-47f8-a8ab-5cf565015621\" (UID: \"19c83f05-c0d9-47f8-a8ab-5cf565015621\") " Nov 02 23:21:48.834489 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:48.834459 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c83f05-c0d9-47f8-a8ab-5cf565015621-openshift-service-ca" pod "19c83f05-c0d9-47f8-a8ab-5cf565015621" (UID: "19c83f05-c0d9-47f8-a8ab-5cf565015621"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:21:48.835023 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:48.834999 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19c83f05-c0d9-47f8-a8ab-5cf565015621-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:48.836313 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:48.836286 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19c83f05-c0d9-47f8-a8ab-5cf565015621-maas-api-tls" pod "19c83f05-c0d9-47f8-a8ab-5cf565015621" (UID: "19c83f05-c0d9-47f8-a8ab-5cf565015621"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:21:48.836795 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:48.836775 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19c83f05-c0d9-47f8-a8ab-5cf565015621-metrics-certs" pod "19c83f05-c0d9-47f8-a8ab-5cf565015621" (UID: "19c83f05-c0d9-47f8-a8ab-5cf565015621"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:21:48.836859 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:48.836800 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c83f05-c0d9-47f8-a8ab-5cf565015621-kube-api-access-98dn5" pod "19c83f05-c0d9-47f8-a8ab-5cf565015621" (UID: "19c83f05-c0d9-47f8-a8ab-5cf565015621"). InnerVolumeSpecName "kube-api-access-98dn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:21:48.936330 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:48.936293 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19c83f05-c0d9-47f8-a8ab-5cf565015621-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:48.936330 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:48.936332 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19c83f05-c0d9-47f8-a8ab-5cf565015621-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:48.936549 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:48.936348 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98dn5\" (UniqueName: \"kubernetes.io/projected/19c83f05-c0d9-47f8-a8ab-5cf565015621-kube-api-access-98dn5\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:49.765880 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:49.765847 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc" Nov 02 23:21:49.931107 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:49.931079 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc"] Nov 02 23:21:49.982414 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:49.982379 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa2454-74df6878bf-d9tsc"] Nov 02 23:21:51.884782 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:51.884740 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19c83f05-c0d9-47f8-a8ab-5cf565015621" path="/var/lib/kubelet/pods/19c83f05-c0d9-47f8-a8ab-5cf565015621/volumes" Nov 02 23:21:52.746436 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:21:52.746406 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9" Nov 02 23:21:57.846389 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:21:57.846348 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:22:27.846880 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:22:27.846844 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:22:32.562529 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:32.562494 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw"] Nov 02 23:22:32.562931 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:32.562711 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" podUID="4cb4d651-3373-4449-8f2f-0cc8f748b1b3" containerName="istio-proxy" containerID="cri-o://3cfc6d5d277674e2c4e964b7b33db50bffe4f2b9460b7b4dedae4c1a52bf2940" gracePeriod=30 Nov 02 23:22:35.480735 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:35.480693 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-gm8cw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:22:35.481233 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:35.480769 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" podUID="4cb4d651-3373-4449-8f2f-0cc8f748b1b3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:22:37.999116 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:37.999088 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:22:38.001397 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.001370 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istiod-ca-cert\") pod \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " Nov 02 23:22:38.001520 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.001413 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-workload-certs\") pod \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " Nov 02 23:22:38.001520 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.001461 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istio-envoy\") pod \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " Nov 02 23:22:38.001520 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.001488 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-credential-socket\") pod \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " Nov 02 23:22:38.001648 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.001538 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istio-data\") pod \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " Nov 02 23:22:38.001648 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.001575 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-workload-socket\") pod \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " Nov 02 23:22:38.001648 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.001597 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-kube-api-access-fx74d\" (UniqueName: \"kubernetes.io/projected/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-kube-api-access-fx74d\") pod \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " Nov 02 23:22:38.001742 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.001712 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istio-podinfo\") pod \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " Nov 02 23:22:38.001776 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.001749 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istio-token\" (UniqueName: \"kubernetes.io/projected/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istio-token\") pod \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\" (UID: \"4cb4d651-3373-4449-8f2f-0cc8f748b1b3\") " Nov 02 23:22:38.002662 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.002626 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istiod-ca-cert" pod "4cb4d651-3373-4449-8f2f-0cc8f748b1b3" (UID: "4cb4d651-3373-4449-8f2f-0cc8f748b1b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:22:38.002837 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.002822 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-workload-certs" pod "4cb4d651-3373-4449-8f2f-0cc8f748b1b3" (UID: "4cb4d651-3373-4449-8f2f-0cc8f748b1b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:38.003474 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.003440 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-workload-socket" pod "4cb4d651-3373-4449-8f2f-0cc8f748b1b3" (UID: "4cb4d651-3373-4449-8f2f-0cc8f748b1b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:38.003693 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.003599 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-credential-socket" pod "4cb4d651-3373-4449-8f2f-0cc8f748b1b3" (UID: "4cb4d651-3373-4449-8f2f-0cc8f748b1b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:38.004022 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.003726 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istio-data" pod "4cb4d651-3373-4449-8f2f-0cc8f748b1b3" (UID: "4cb4d651-3373-4449-8f2f-0cc8f748b1b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:38.005910 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.005888 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istio-podinfo" pod "4cb4d651-3373-4449-8f2f-0cc8f748b1b3" (UID: "4cb4d651-3373-4449-8f2f-0cc8f748b1b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:22:38.006256 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.006234 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-kube-api-access-fx74d" pod "4cb4d651-3373-4449-8f2f-0cc8f748b1b3" (UID: "4cb4d651-3373-4449-8f2f-0cc8f748b1b3"). InnerVolumeSpecName "kube-api-access-fx74d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:38.006463 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.006442 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istio-envoy" pod "4cb4d651-3373-4449-8f2f-0cc8f748b1b3" (UID: "4cb4d651-3373-4449-8f2f-0cc8f748b1b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:38.010495 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.010470 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istio-token" pod "4cb4d651-3373-4449-8f2f-0cc8f748b1b3" (UID: "4cb4d651-3373-4449-8f2f-0cc8f748b1b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:38.103749 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.103711 5727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istio-podinfo\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:38.103749 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.103742 5727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istio-token\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:38.103749 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.103755 5727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istiod-ca-cert\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:38.103990 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.103768 5727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-workload-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:38.103990 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.103782 5727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istio-envoy\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:38.103990 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.103794 5727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-credential-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:38.103990 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.103804 5727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-istio-data\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:38.103990 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.103818 5727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-workload-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:38.103990 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.103830 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx74d\" (UniqueName: \"kubernetes.io/projected/4cb4d651-3373-4449-8f2f-0cc8f748b1b3-kube-api-access-fx74d\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:38.174467 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.174431 5727 generic.go:356] "Generic (PLEG): container finished" podID="4cb4d651-3373-4449-8f2f-0cc8f748b1b3" containerID="3cfc6d5d277674e2c4e964b7b33db50bffe4f2b9460b7b4dedae4c1a52bf2940" exitCode=0 Nov 02 23:22:38.174714 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.174468 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" event={"ID":"4cb4d651-3373-4449-8f2f-0cc8f748b1b3","Type":"ContainerDied","Data":"3cfc6d5d277674e2c4e964b7b33db50bffe4f2b9460b7b4dedae4c1a52bf2940"} Nov 02 23:22:38.174714 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.174544 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" event={"ID":"4cb4d651-3373-4449-8f2f-0cc8f748b1b3","Type":"ContainerDied","Data":"7e47130369c970d5cd76d2851e0832c059ec3b2e40bd5dedddc3d484b5597ec2"} Nov 02 23:22:38.174714 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.174577 5727 scope.go:122] "RemoveContainer" containerID="3cfc6d5d277674e2c4e964b7b33db50bffe4f2b9460b7b4dedae4c1a52bf2940" Nov 02 23:22:38.176214 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.176190 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw" Nov 02 23:22:38.191340 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.191321 5727 scope.go:122] "RemoveContainer" containerID="3cfc6d5d277674e2c4e964b7b33db50bffe4f2b9460b7b4dedae4c1a52bf2940" Nov 02 23:22:38.191765 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:22:38.191723 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cfc6d5d277674e2c4e964b7b33db50bffe4f2b9460b7b4dedae4c1a52bf2940\": container with ID starting with 3cfc6d5d277674e2c4e964b7b33db50bffe4f2b9460b7b4dedae4c1a52bf2940 not found: ID does not exist" containerID="3cfc6d5d277674e2c4e964b7b33db50bffe4f2b9460b7b4dedae4c1a52bf2940" Nov 02 23:22:38.191915 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.191759 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cfc6d5d277674e2c4e964b7b33db50bffe4f2b9460b7b4dedae4c1a52bf2940"} err="failed to get container status \"3cfc6d5d277674e2c4e964b7b33db50bffe4f2b9460b7b4dedae4c1a52bf2940\": rpc error: code = NotFound desc = could not find container \"3cfc6d5d277674e2c4e964b7b33db50bffe4f2b9460b7b4dedae4c1a52bf2940\": container with ID starting with 3cfc6d5d277674e2c4e964b7b33db50bffe4f2b9460b7b4dedae4c1a52bf2940 not found: ID does not exist" Nov 02 23:22:38.342483 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.342453 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw"] Nov 02 23:22:38.346019 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:38.345999 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gm8cw"] Nov 02 23:22:39.463210 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:39.463173 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r"] Nov 02 23:22:39.463643 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:39.463425 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" podUID="2ae279e6-131e-446b-9ad5-6c69582fa70f" containerName="istio-proxy" containerID="cri-o://311762f5998ee29aa53844c2e13dbe3bc0c8cf788bd7ca0074629d72e4507cf4" gracePeriod=30 Nov 02 23:22:39.885912 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:39.885877 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cb4d651-3373-4449-8f2f-0cc8f748b1b3" path="/var/lib/kubelet/pods/4cb4d651-3373-4449-8f2f-0cc8f748b1b3/volumes" Nov 02 23:22:41.624497 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:41.624462 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:22:41.624939 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:41.624520 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" podUID="2ae279e6-131e-446b-9ad5-6c69582fa70f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:22:44.740591 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:44.740531 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:22:44.784181 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:44.784116 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-credential-socket\") pod \"2ae279e6-131e-446b-9ad5-6c69582fa70f\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " Nov 02 23:22:44.784353 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:44.784254 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-istio-envoy\") pod \"2ae279e6-131e-446b-9ad5-6c69582fa70f\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " Nov 02 23:22:44.784353 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:44.784298 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae279e6-131e-446b-9ad5-6c69582fa70f-istio-token\" (UniqueName: \"kubernetes.io/projected/2ae279e6-131e-446b-9ad5-6c69582fa70f-istio-token\") pod \"2ae279e6-131e-446b-9ad5-6c69582fa70f\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " Nov 02 23:22:44.784488 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:44.784363 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-workload-socket\") pod \"2ae279e6-131e-446b-9ad5-6c69582fa70f\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " Nov 02 23:22:44.784488 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:44.784377 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-credential-socket" pod "2ae279e6-131e-446b-9ad5-6c69582fa70f" (UID: "2ae279e6-131e-446b-9ad5-6c69582fa70f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.784488 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:44.784395 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-istio-data\") pod \"2ae279e6-131e-446b-9ad5-6c69582fa70f\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " Nov 02 23:22:44.784488 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:44.784464 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ae279e6-131e-446b-9ad5-6c69582fa70f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae279e6-131e-446b-9ad5-6c69582fa70f-istiod-ca-cert\") pod \"2ae279e6-131e-446b-9ad5-6c69582fa70f\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " Nov 02 23:22:44.784686 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:44.784581 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae279e6-131e-446b-9ad5-6c69582fa70f-kube-api-access-hjqtt\" (UniqueName: \"kubernetes.io/projected/2ae279e6-131e-446b-9ad5-6c69582fa70f-kube-api-access-hjqtt\") pod \"2ae279e6-131e-446b-9ad5-6c69582fa70f\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " Nov 02 23:22:44.784686 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:44.784639 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-workload-certs\") pod \"2ae279e6-131e-446b-9ad5-6c69582fa70f\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " Nov 02 23:22:44.784686 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:44.784656 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-workload-socket" pod "2ae279e6-131e-446b-9ad5-6c69582fa70f" (UID: "2ae279e6-131e-446b-9ad5-6c69582fa70f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.784686 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:44.784680 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ae279e6-131e-446b-9ad5-6c69582fa70f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae279e6-131e-446b-9ad5-6c69582fa70f-istio-podinfo\") pod \"2ae279e6-131e-446b-9ad5-6c69582fa70f\" (UID: \"2ae279e6-131e-446b-9ad5-6c69582fa70f\") " Nov 02 23:22:44.784903 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:44.784681 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-istio-data" pod "2ae279e6-131e-446b-9ad5-6c69582fa70f" (UID: "2ae279e6-131e-446b-9ad5-6c69582fa70f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.784960 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:44.784901 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-workload-certs" pod "2ae279e6-131e-446b-9ad5-6c69582fa70f" (UID: "2ae279e6-131e-446b-9ad5-6c69582fa70f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.784960 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:44.784933 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae279e6-131e-446b-9ad5-6c69582fa70f-istiod-ca-cert" pod "2ae279e6-131e-446b-9ad5-6c69582fa70f" (UID: "2ae279e6-131e-446b-9ad5-6c69582fa70f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:22:44.785288 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:44.785268 5727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-credential-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.785375 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:44.785291 5727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-workload-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.785375 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:44.785306 5727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-istio-data\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.785375 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:44.785318 5727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae279e6-131e-446b-9ad5-6c69582fa70f-istiod-ca-cert\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.785375 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:44.785331 5727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-workload-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.787089 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:44.787066 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae279e6-131e-446b-9ad5-6c69582fa70f-istio-token" pod "2ae279e6-131e-446b-9ad5-6c69582fa70f" (UID: "2ae279e6-131e-446b-9ad5-6c69582fa70f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:44.787300 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:44.787253 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae279e6-131e-446b-9ad5-6c69582fa70f-kube-api-access-hjqtt" pod "2ae279e6-131e-446b-9ad5-6c69582fa70f" (UID: "2ae279e6-131e-446b-9ad5-6c69582fa70f"). InnerVolumeSpecName "kube-api-access-hjqtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:44.787367 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:44.787299 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-istio-envoy" pod "2ae279e6-131e-446b-9ad5-6c69582fa70f" (UID: "2ae279e6-131e-446b-9ad5-6c69582fa70f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.787526 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:44.787503 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ae279e6-131e-446b-9ad5-6c69582fa70f-istio-podinfo" pod "2ae279e6-131e-446b-9ad5-6c69582fa70f" (UID: "2ae279e6-131e-446b-9ad5-6c69582fa70f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:22:44.886550 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:44.886523 5727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae279e6-131e-446b-9ad5-6c69582fa70f-istio-podinfo\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.886550 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:44.886548 5727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae279e6-131e-446b-9ad5-6c69582fa70f-istio-envoy\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.886550 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:44.886557 5727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae279e6-131e-446b-9ad5-6c69582fa70f-istio-token\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.886756 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:44.886566 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjqtt\" (UniqueName: \"kubernetes.io/projected/2ae279e6-131e-446b-9ad5-6c69582fa70f-kube-api-access-hjqtt\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:45.242524 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:45.242489 5727 generic.go:356] "Generic (PLEG): container finished" podID="2ae279e6-131e-446b-9ad5-6c69582fa70f" containerID="311762f5998ee29aa53844c2e13dbe3bc0c8cf788bd7ca0074629d72e4507cf4" exitCode=0 Nov 02 23:22:45.242669 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:45.242542 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" Nov 02 23:22:45.242669 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:45.242569 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" event={"ID":"2ae279e6-131e-446b-9ad5-6c69582fa70f","Type":"ContainerDied","Data":"311762f5998ee29aa53844c2e13dbe3bc0c8cf788bd7ca0074629d72e4507cf4"} Nov 02 23:22:45.242669 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:45.242633 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r" event={"ID":"2ae279e6-131e-446b-9ad5-6c69582fa70f","Type":"ContainerDied","Data":"8b856742f0ce9357283a5094ad62a86cd5ea23156e0dfcd17ff3a52e7b9746c2"} Nov 02 23:22:45.242669 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:45.242659 5727 scope.go:122] "RemoveContainer" containerID="311762f5998ee29aa53844c2e13dbe3bc0c8cf788bd7ca0074629d72e4507cf4" Nov 02 23:22:45.255314 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:45.255295 5727 scope.go:122] "RemoveContainer" containerID="311762f5998ee29aa53844c2e13dbe3bc0c8cf788bd7ca0074629d72e4507cf4" Nov 02 23:22:45.255668 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:22:45.255641 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"311762f5998ee29aa53844c2e13dbe3bc0c8cf788bd7ca0074629d72e4507cf4\": container with ID starting with 311762f5998ee29aa53844c2e13dbe3bc0c8cf788bd7ca0074629d72e4507cf4 not found: ID does not exist" containerID="311762f5998ee29aa53844c2e13dbe3bc0c8cf788bd7ca0074629d72e4507cf4" Nov 02 23:22:45.255790 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:45.255679 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"311762f5998ee29aa53844c2e13dbe3bc0c8cf788bd7ca0074629d72e4507cf4"} err="failed to get container status \"311762f5998ee29aa53844c2e13dbe3bc0c8cf788bd7ca0074629d72e4507cf4\": rpc error: code = NotFound desc = could not find container \"311762f5998ee29aa53844c2e13dbe3bc0c8cf788bd7ca0074629d72e4507cf4\": container with ID starting with 311762f5998ee29aa53844c2e13dbe3bc0c8cf788bd7ca0074629d72e4507cf4 not found: ID does not exist" Nov 02 23:22:45.412725 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:45.411761 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r"] Nov 02 23:22:45.419363 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:45.419328 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-vsm5r"] Nov 02 23:22:45.468862 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:45.468831 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e875bc-5f4d6557b7-ct7tk"] Nov 02 23:22:45.469450 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:45.469437 5727 cpu_manager.go:414] "containerMap: removing container" podUID="2ae279e6-131e-446b-9ad5-6c69582fa70f" containerName="istio-proxy" Nov 02 23:22:45.469508 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:45.469451 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae279e6-131e-446b-9ad5-6c69582fa70f" containerName="istio-proxy" Nov 02 23:22:45.469508 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:45.469478 5727 cpu_manager.go:414] "containerMap: removing container" podUID="19c83f05-c0d9-47f8-a8ab-5cf565015621" containerName="maas-api" Nov 02 23:22:45.469508 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:45.469483 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19c83f05-c0d9-47f8-a8ab-5cf565015621" containerName="maas-api" Nov 02 23:22:45.469508 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:45.469493 5727 cpu_manager.go:414] "containerMap: removing container" podUID="4cb4d651-3373-4449-8f2f-0cc8f748b1b3" containerName="istio-proxy" Nov 02 23:22:45.469508 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:45.469499 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cb4d651-3373-4449-8f2f-0cc8f748b1b3" containerName="istio-proxy" Nov 02 23:22:45.469658 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:45.469602 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="19c83f05-c0d9-47f8-a8ab-5cf565015621" containerName="maas-api" Nov 02 23:22:45.469658 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:45.469611 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cb4d651-3373-4449-8f2f-0cc8f748b1b3" containerName="istio-proxy" Nov 02 23:22:45.469658 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:45.469622 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae279e6-131e-446b-9ad5-6c69582fa70f" containerName="istio-proxy" Nov 02 23:22:45.474401 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:45.474375 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e875bc-5f4d6557b7-ct7tk" Nov 02 23:22:45.526690 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:45.526628 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-e875bc-dockercfg-m8pl8\"" Nov 02 23:22:45.526902 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:45.526884 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-e875bc\"" Nov 02 23:22:45.529716 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:45.529696 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e875bc-5f4d6557b7-ct7tk"] Nov 02 23:22:45.611152 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:45.607235 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5zk9\" (UniqueName: \"kubernetes.io/projected/fde5fb1f-bd36-45b0-83fa-b3be9e144340-kube-api-access-v5zk9\") pod \"payload-pre-processing-e2e-api-b-e875bc-5f4d6557b7-ct7tk\" (UID: \"fde5fb1f-bd36-45b0-83fa-b3be9e144340\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e875bc-5f4d6557b7-ct7tk" Nov 02 23:22:45.611152 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:45.607278 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fde5fb1f-bd36-45b0-83fa-b3be9e144340-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e875bc-5f4d6557b7-ct7tk\" (UID: \"fde5fb1f-bd36-45b0-83fa-b3be9e144340\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e875bc-5f4d6557b7-ct7tk" Nov 02 23:22:45.708188 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:45.708149 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5zk9\" (UniqueName: \"kubernetes.io/projected/fde5fb1f-bd36-45b0-83fa-b3be9e144340-kube-api-access-v5zk9\") pod \"payload-pre-processing-e2e-api-b-e875bc-5f4d6557b7-ct7tk\" (UID: \"fde5fb1f-bd36-45b0-83fa-b3be9e144340\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e875bc-5f4d6557b7-ct7tk" Nov 02 23:22:45.708379 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:45.708201 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fde5fb1f-bd36-45b0-83fa-b3be9e144340-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e875bc-5f4d6557b7-ct7tk\" (UID: \"fde5fb1f-bd36-45b0-83fa-b3be9e144340\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e875bc-5f4d6557b7-ct7tk" Nov 02 23:22:45.709269 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:45.709247 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fde5fb1f-bd36-45b0-83fa-b3be9e144340-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e875bc-5f4d6557b7-ct7tk\" (UID: \"fde5fb1f-bd36-45b0-83fa-b3be9e144340\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e875bc-5f4d6557b7-ct7tk" Nov 02 23:22:45.769877 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:45.769845 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5zk9\" (UniqueName: \"kubernetes.io/projected/fde5fb1f-bd36-45b0-83fa-b3be9e144340-kube-api-access-v5zk9\") pod \"payload-pre-processing-e2e-api-b-e875bc-5f4d6557b7-ct7tk\" (UID: \"fde5fb1f-bd36-45b0-83fa-b3be9e144340\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e875bc-5f4d6557b7-ct7tk" Nov 02 23:22:45.789971 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:45.789911 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e875bc-5f4d6557b7-ct7tk" Nov 02 23:22:45.895762 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:45.895718 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ae279e6-131e-446b-9ad5-6c69582fa70f" path="/var/lib/kubelet/pods/2ae279e6-131e-446b-9ad5-6c69582fa70f/volumes" Nov 02 23:22:46.166813 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:46.166787 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e875bc-5f4d6557b7-ct7tk"] Nov 02 23:22:46.168031 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:22:46.168002 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfde5fb1f_bd36_45b0_83fa_b3be9e144340.slice/crio-f009efd9b8f85feecf812fc12db950b3dea39c40024b9823f4d3bf0d12c1c106 WatchSource:0}: Error finding container f009efd9b8f85feecf812fc12db950b3dea39c40024b9823f4d3bf0d12c1c106: Status 404 returned error can't find the container with id f009efd9b8f85feecf812fc12db950b3dea39c40024b9823f4d3bf0d12c1c106 Nov 02 23:22:46.251352 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:46.251322 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e875bc-5f4d6557b7-ct7tk" event={"ID":"fde5fb1f-bd36-45b0-83fa-b3be9e144340","Type":"ContainerStarted","Data":"ee2244f3b868019d45e238968ba5efea1beaf13a617c7efa8d75926c558f3dcd"} Nov 02 23:22:46.251492 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:46.251382 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e875bc-5f4d6557b7-ct7tk" event={"ID":"fde5fb1f-bd36-45b0-83fa-b3be9e144340","Type":"ContainerStarted","Data":"f009efd9b8f85feecf812fc12db950b3dea39c40024b9823f4d3bf0d12c1c106"} Nov 02 23:22:46.251492 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:46.251427 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e875bc-5f4d6557b7-ct7tk" Nov 02 23:22:46.367040 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:46.366973 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e875bc-5f4d6557b7-ct7tk" podStartSLOduration=1.366955779 podStartE2EDuration="1.366955779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:22:46.364949917 +0000 UTC m=+2052.603447581" watchObservedRunningTime="2026-11-02 23:22:46.366955779 +0000 UTC m=+2052.605453442" Nov 02 23:22:57.274191 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:57.274165 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e875bc-5f4d6557b7-ct7tk" Nov 02 23:22:57.846439 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:22:57.846403 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:22:58.811218 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:58.811186 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-92072f-5bf4fb99-kn2nd"] Nov 02 23:22:58.919234 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:22:58.919194 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-6844b65d7-d9v9r"] Nov 02 23:23:03.812420 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:03.812360 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-5bf4fb99-kn2nd" podUID="3ac2e404-7f02-4318-9aaf-0c33d1e99124" containerName="payload-processing" containerID="cri-o://0b687f022d87bb3de9aaf69521a5cf9e44f8472e88e15f76c6fd22365aefa79f" gracePeriod=25 Nov 02 23:23:03.920210 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:03.920116 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-6844b65d7-d9v9r" podUID="c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa" containerName="payload-pre-processing" containerID="cri-o://71882a6d1a3ec3bdb70bc44fc49948a80d9ece6a478ce040fa031c12f6e325fe" gracePeriod=25 Nov 02 23:23:04.104311 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:04.104283 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-5bf4fb99-kn2nd" Nov 02 23:23:04.268363 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:04.231377 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ac2e404-7f02-4318-9aaf-0c33d1e99124-kube-api-access-xx8qf\" (UniqueName: \"kubernetes.io/projected/3ac2e404-7f02-4318-9aaf-0c33d1e99124-kube-api-access-xx8qf\") pod \"3ac2e404-7f02-4318-9aaf-0c33d1e99124\" (UID: \"3ac2e404-7f02-4318-9aaf-0c33d1e99124\") " Nov 02 23:23:04.268363 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:04.239408 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ac2e404-7f02-4318-9aaf-0c33d1e99124-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ac2e404-7f02-4318-9aaf-0c33d1e99124-plugins-config-volume\") pod \"3ac2e404-7f02-4318-9aaf-0c33d1e99124\" (UID: \"3ac2e404-7f02-4318-9aaf-0c33d1e99124\") " Nov 02 23:23:04.268363 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:04.262373 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ac2e404-7f02-4318-9aaf-0c33d1e99124-plugins-config-volume" pod "3ac2e404-7f02-4318-9aaf-0c33d1e99124" (UID: "3ac2e404-7f02-4318-9aaf-0c33d1e99124"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:23:04.319375 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:04.319320 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac2e404-7f02-4318-9aaf-0c33d1e99124-kube-api-access-xx8qf" pod "3ac2e404-7f02-4318-9aaf-0c33d1e99124" (UID: "3ac2e404-7f02-4318-9aaf-0c33d1e99124"). InnerVolumeSpecName "kube-api-access-xx8qf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:23:04.335053 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:04.334979 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-6844b65d7-d9v9r" Nov 02 23:23:04.340938 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:04.340912 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx8qf\" (UniqueName: \"kubernetes.io/projected/3ac2e404-7f02-4318-9aaf-0c33d1e99124-kube-api-access-xx8qf\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:23:04.340938 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:04.340937 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ac2e404-7f02-4318-9aaf-0c33d1e99124-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:23:04.442160 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:04.442108 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa-plugins-config-volume\") pod \"c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa\" (UID: \"c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa\") " Nov 02 23:23:04.442344 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:04.442238 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa-kube-api-access-qw7bh\" (UniqueName: \"kubernetes.io/projected/c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa-kube-api-access-qw7bh\") pod \"c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa\" (UID: \"c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa\") " Nov 02 23:23:04.442456 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:04.442434 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa-plugins-config-volume" pod "c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa" (UID: "c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:23:04.442698 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:04.442679 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:23:04.444489 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:04.444462 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa-kube-api-access-qw7bh" pod "c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa" (UID: "c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa"). InnerVolumeSpecName "kube-api-access-qw7bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:23:04.490597 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:04.490558 5727 generic.go:356] "Generic (PLEG): container finished" podID="c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa" containerID="71882a6d1a3ec3bdb70bc44fc49948a80d9ece6a478ce040fa031c12f6e325fe" exitCode=0 Nov 02 23:23:04.491002 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:04.490611 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-6844b65d7-d9v9r" Nov 02 23:23:04.491002 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:04.490642 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-6844b65d7-d9v9r" event={"ID":"c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa","Type":"ContainerDied","Data":"71882a6d1a3ec3bdb70bc44fc49948a80d9ece6a478ce040fa031c12f6e325fe"} Nov 02 23:23:04.491253 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:04.491020 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-6844b65d7-d9v9r" event={"ID":"c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa","Type":"ContainerDied","Data":"d9eccd301426fc5238ff29cf3dd2650337834aa972a9b6c87dbfb80908cb3afc"} Nov 02 23:23:04.491253 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:04.491052 5727 scope.go:122] "RemoveContainer" containerID="71882a6d1a3ec3bdb70bc44fc49948a80d9ece6a478ce040fa031c12f6e325fe" Nov 02 23:23:04.497455 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:04.497425 5727 generic.go:356] "Generic (PLEG): container finished" podID="3ac2e404-7f02-4318-9aaf-0c33d1e99124" containerID="0b687f022d87bb3de9aaf69521a5cf9e44f8472e88e15f76c6fd22365aefa79f" exitCode=0 Nov 02 23:23:04.497678 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:04.497661 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-5bf4fb99-kn2nd" event={"ID":"3ac2e404-7f02-4318-9aaf-0c33d1e99124","Type":"ContainerDied","Data":"0b687f022d87bb3de9aaf69521a5cf9e44f8472e88e15f76c6fd22365aefa79f"} Nov 02 23:23:04.497756 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:04.497726 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-5bf4fb99-kn2nd" event={"ID":"3ac2e404-7f02-4318-9aaf-0c33d1e99124","Type":"ContainerDied","Data":"397afb4656ca8d6b04c82bf58019971db8e48afcdcd39cf485c4c73a189c7fd3"} Nov 02 23:23:04.497856 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:04.497829 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-5bf4fb99-kn2nd" Nov 02 23:23:04.508506 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:04.508484 5727 scope.go:122] "RemoveContainer" containerID="71882a6d1a3ec3bdb70bc44fc49948a80d9ece6a478ce040fa031c12f6e325fe" Nov 02 23:23:04.508827 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:23:04.508805 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71882a6d1a3ec3bdb70bc44fc49948a80d9ece6a478ce040fa031c12f6e325fe\": container with ID starting with 71882a6d1a3ec3bdb70bc44fc49948a80d9ece6a478ce040fa031c12f6e325fe not found: ID does not exist" containerID="71882a6d1a3ec3bdb70bc44fc49948a80d9ece6a478ce040fa031c12f6e325fe" Nov 02 23:23:04.508827 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:04.508832 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71882a6d1a3ec3bdb70bc44fc49948a80d9ece6a478ce040fa031c12f6e325fe"} err="failed to get container status \"71882a6d1a3ec3bdb70bc44fc49948a80d9ece6a478ce040fa031c12f6e325fe\": rpc error: code = NotFound desc = could not find container \"71882a6d1a3ec3bdb70bc44fc49948a80d9ece6a478ce040fa031c12f6e325fe\": container with ID starting with 71882a6d1a3ec3bdb70bc44fc49948a80d9ece6a478ce040fa031c12f6e325fe not found: ID does not exist" Nov 02 23:23:04.508940 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:04.508848 5727 scope.go:122] "RemoveContainer" containerID="0b687f022d87bb3de9aaf69521a5cf9e44f8472e88e15f76c6fd22365aefa79f" Nov 02 23:23:04.518962 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:04.518946 5727 scope.go:122] "RemoveContainer" containerID="0b687f022d87bb3de9aaf69521a5cf9e44f8472e88e15f76c6fd22365aefa79f" Nov 02 23:23:04.519239 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:23:04.519223 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b687f022d87bb3de9aaf69521a5cf9e44f8472e88e15f76c6fd22365aefa79f\": container with ID starting with 0b687f022d87bb3de9aaf69521a5cf9e44f8472e88e15f76c6fd22365aefa79f not found: ID does not exist" containerID="0b687f022d87bb3de9aaf69521a5cf9e44f8472e88e15f76c6fd22365aefa79f" Nov 02 23:23:04.519290 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:04.519246 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b687f022d87bb3de9aaf69521a5cf9e44f8472e88e15f76c6fd22365aefa79f"} err="failed to get container status \"0b687f022d87bb3de9aaf69521a5cf9e44f8472e88e15f76c6fd22365aefa79f\": rpc error: code = NotFound desc = could not find container \"0b687f022d87bb3de9aaf69521a5cf9e44f8472e88e15f76c6fd22365aefa79f\": container with ID starting with 0b687f022d87bb3de9aaf69521a5cf9e44f8472e88e15f76c6fd22365aefa79f not found: ID does not exist" Nov 02 23:23:04.543585 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:04.543565 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw7bh\" (UniqueName: \"kubernetes.io/projected/c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa-kube-api-access-qw7bh\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:23:04.661637 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:04.661555 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-92072f-5bf4fb99-kn2nd"] Nov 02 23:23:04.666859 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:04.666830 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-92072f-5bf4fb99-kn2nd"] Nov 02 23:23:04.772336 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:04.772303 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-6844b65d7-d9v9r"] Nov 02 23:23:04.781936 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:04.781911 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-6844b65d7-d9v9r"] Nov 02 23:23:05.885522 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:05.885483 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ac2e404-7f02-4318-9aaf-0c33d1e99124" path="/var/lib/kubelet/pods/3ac2e404-7f02-4318-9aaf-0c33d1e99124/volumes" Nov 02 23:23:05.886032 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:05.886007 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa" path="/var/lib/kubelet/pods/c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa/volumes" Nov 02 23:23:27.847193 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:23:27.847120 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:23:34.439997 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:34.439964 5727 scope.go:122] "RemoveContainer" containerID="824526bfec4f627dc093d6558f6dd3a03cc5849bd3d7de7482644a80c416b842" Nov 02 23:23:34.456491 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:23:34.456456 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"824526bfec4f627dc093d6558f6dd3a03cc5849bd3d7de7482644a80c416b842\": container with ID starting with 824526bfec4f627dc093d6558f6dd3a03cc5849bd3d7de7482644a80c416b842 not found: ID does not exist" containerID="824526bfec4f627dc093d6558f6dd3a03cc5849bd3d7de7482644a80c416b842" Nov 02 23:23:34.456640 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:34.456504 5727 image_gc_manager.go:223] "Failed to monitor images" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"824526bfec4f627dc093d6558f6dd3a03cc5849bd3d7de7482644a80c416b842\": container with ID starting with 824526bfec4f627dc093d6558f6dd3a03cc5849bd3d7de7482644a80c416b842 not found: ID does not exist" Nov 02 23:23:34.471307 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:34.471229 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 02 23:23:34.518779 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:34.518759 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 23:23:35.914859 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:35.914824 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-5674cb547f-cgtjh"] Nov 02 23:23:35.915526 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:35.915509 5727 cpu_manager.go:414] "containerMap: removing container" podUID="c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa" containerName="payload-pre-processing" Nov 02 23:23:35.915572 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:35.915527 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa" containerName="payload-pre-processing" Nov 02 23:23:35.915572 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:35.915551 5727 cpu_manager.go:414] "containerMap: removing container" podUID="3ac2e404-7f02-4318-9aaf-0c33d1e99124" containerName="payload-processing" Nov 02 23:23:35.915572 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:35.915556 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ac2e404-7f02-4318-9aaf-0c33d1e99124" containerName="payload-processing" Nov 02 23:23:35.915723 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:35.915715 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="c703dc6f-7178-43b9-b5fa-9de1e1b9c2fa" containerName="payload-pre-processing" Nov 02 23:23:35.915763 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:35.915727 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ac2e404-7f02-4318-9aaf-0c33d1e99124" containerName="payload-processing" Nov 02 23:23:35.920931 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:35.920911 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-5674cb547f-cgtjh" Nov 02 23:23:35.978017 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:35.977984 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-5674cb547f-cgtjh"] Nov 02 23:23:36.021664 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:36.021636 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-014ab4\"" Nov 02 23:23:36.100607 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:36.100569 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzq6k\" (UniqueName: \"kubernetes.io/projected/63e2eb91-30e8-49e7-8b0a-9a66435e46e8-kube-api-access-fzq6k\") pod \"payload-pre-processing-e2e-ipp-b-014ab4-5674cb547f-cgtjh\" (UID: \"63e2eb91-30e8-49e7-8b0a-9a66435e46e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-5674cb547f-cgtjh" Nov 02 23:23:36.100875 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:36.100856 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63e2eb91-30e8-49e7-8b0a-9a66435e46e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-014ab4-5674cb547f-cgtjh\" (UID: \"63e2eb91-30e8-49e7-8b0a-9a66435e46e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-5674cb547f-cgtjh" Nov 02 23:23:36.119213 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:36.110438 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7549c7cff-j4lm4"] Nov 02 23:23:36.119213 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:36.116794 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7549c7cff-j4lm4" Nov 02 23:23:36.136444 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:36.136415 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7549c7cff-j4lm4"] Nov 02 23:23:36.169808 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:36.169723 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-014ab4-dockercfg-jqf2v\"" Nov 02 23:23:36.203632 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:36.203600 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbvxv\" (UniqueName: \"kubernetes.io/projected/ea5fa62a-c0d2-412c-ac94-5b5f4482b110-kube-api-access-rbvxv\") pod \"payload-processing-e2e-ipp-b-014ab4-7549c7cff-j4lm4\" (UID: \"ea5fa62a-c0d2-412c-ac94-5b5f4482b110\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7549c7cff-j4lm4" Nov 02 23:23:36.203822 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:36.203668 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5fa62a-c0d2-412c-ac94-5b5f4482b110-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-014ab4-7549c7cff-j4lm4\" (UID: \"ea5fa62a-c0d2-412c-ac94-5b5f4482b110\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7549c7cff-j4lm4" Nov 02 23:23:36.203822 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:36.203768 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzq6k\" (UniqueName: \"kubernetes.io/projected/63e2eb91-30e8-49e7-8b0a-9a66435e46e8-kube-api-access-fzq6k\") pod \"payload-pre-processing-e2e-ipp-b-014ab4-5674cb547f-cgtjh\" (UID: \"63e2eb91-30e8-49e7-8b0a-9a66435e46e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-5674cb547f-cgtjh" Nov 02 23:23:36.203822 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:36.203800 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63e2eb91-30e8-49e7-8b0a-9a66435e46e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-014ab4-5674cb547f-cgtjh\" (UID: \"63e2eb91-30e8-49e7-8b0a-9a66435e46e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-5674cb547f-cgtjh" Nov 02 23:23:36.204741 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:36.204715 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63e2eb91-30e8-49e7-8b0a-9a66435e46e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-014ab4-5674cb547f-cgtjh\" (UID: \"63e2eb91-30e8-49e7-8b0a-9a66435e46e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-5674cb547f-cgtjh" Nov 02 23:23:36.264539 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:36.264505 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzq6k\" (UniqueName: \"kubernetes.io/projected/63e2eb91-30e8-49e7-8b0a-9a66435e46e8-kube-api-access-fzq6k\") pod \"payload-pre-processing-e2e-ipp-b-014ab4-5674cb547f-cgtjh\" (UID: \"63e2eb91-30e8-49e7-8b0a-9a66435e46e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-5674cb547f-cgtjh" Nov 02 23:23:36.304801 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:36.304765 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbvxv\" (UniqueName: \"kubernetes.io/projected/ea5fa62a-c0d2-412c-ac94-5b5f4482b110-kube-api-access-rbvxv\") pod \"payload-processing-e2e-ipp-b-014ab4-7549c7cff-j4lm4\" (UID: \"ea5fa62a-c0d2-412c-ac94-5b5f4482b110\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7549c7cff-j4lm4" Nov 02 23:23:36.304989 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:36.304867 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5fa62a-c0d2-412c-ac94-5b5f4482b110-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-014ab4-7549c7cff-j4lm4\" (UID: \"ea5fa62a-c0d2-412c-ac94-5b5f4482b110\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7549c7cff-j4lm4" Nov 02 23:23:36.305595 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:36.305573 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5fa62a-c0d2-412c-ac94-5b5f4482b110-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-014ab4-7549c7cff-j4lm4\" (UID: \"ea5fa62a-c0d2-412c-ac94-5b5f4482b110\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7549c7cff-j4lm4" Nov 02 23:23:36.365463 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:36.365432 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbvxv\" (UniqueName: \"kubernetes.io/projected/ea5fa62a-c0d2-412c-ac94-5b5f4482b110-kube-api-access-rbvxv\") pod \"payload-processing-e2e-ipp-b-014ab4-7549c7cff-j4lm4\" (UID: \"ea5fa62a-c0d2-412c-ac94-5b5f4482b110\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7549c7cff-j4lm4" Nov 02 23:23:36.447836 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:36.447755 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7549c7cff-j4lm4" Nov 02 23:23:36.536174 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:36.535174 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-5674cb547f-cgtjh" Nov 02 23:23:36.840787 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:36.840762 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7549c7cff-j4lm4"] Nov 02 23:23:36.841679 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:23:36.841650 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5fa62a_c0d2_412c_ac94_5b5f4482b110.slice/crio-1ddb372f0a09631e3d063c8167be4c4550080491dfee5fc6b8abed85af7d82df WatchSource:0}: Error finding container 1ddb372f0a09631e3d063c8167be4c4550080491dfee5fc6b8abed85af7d82df: Status 404 returned error can't find the container with id 1ddb372f0a09631e3d063c8167be4c4550080491dfee5fc6b8abed85af7d82df Nov 02 23:23:36.953165 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:36.953115 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-5674cb547f-cgtjh"] Nov 02 23:23:36.957678 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:23:36.957649 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63e2eb91_30e8_49e7_8b0a_9a66435e46e8.slice/crio-37fd273988d326f8611c4677c95c92b67b499cd097c2afb52bd2cd5dc4242311 WatchSource:0}: Error finding container 37fd273988d326f8611c4677c95c92b67b499cd097c2afb52bd2cd5dc4242311: Status 404 returned error can't find the container with id 37fd273988d326f8611c4677c95c92b67b499cd097c2afb52bd2cd5dc4242311 Nov 02 23:23:37.773947 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:37.773909 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7549c7cff-j4lm4" event={"ID":"ea5fa62a-c0d2-412c-ac94-5b5f4482b110","Type":"ContainerStarted","Data":"01231fb3643cf7a094cc482b55f1966e8b3dc032644203baa597809484081e7f"} Nov 02 23:23:37.774175 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:37.773977 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7549c7cff-j4lm4" event={"ID":"ea5fa62a-c0d2-412c-ac94-5b5f4482b110","Type":"ContainerStarted","Data":"1ddb372f0a09631e3d063c8167be4c4550080491dfee5fc6b8abed85af7d82df"} Nov 02 23:23:37.774175 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:37.774087 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7549c7cff-j4lm4" Nov 02 23:23:37.775748 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:37.775723 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-5674cb547f-cgtjh" event={"ID":"63e2eb91-30e8-49e7-8b0a-9a66435e46e8","Type":"ContainerStarted","Data":"bf5ddc9a741fa2b1010aa119aa19c828bd94bdb55cc2006a5fb6cb5d2ac2657b"} Nov 02 23:23:37.775856 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:37.775770 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-5674cb547f-cgtjh" event={"ID":"63e2eb91-30e8-49e7-8b0a-9a66435e46e8","Type":"ContainerStarted","Data":"37fd273988d326f8611c4677c95c92b67b499cd097c2afb52bd2cd5dc4242311"} Nov 02 23:23:37.775925 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:37.775865 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-5674cb547f-cgtjh" Nov 02 23:23:37.893299 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:37.893253 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7549c7cff-j4lm4" podStartSLOduration=1.8932392519999999 podStartE2EDuration="1.893239252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:23:37.889810716 +0000 UTC m=+2104.128308393" watchObservedRunningTime="2026-11-02 23:23:37.893239252 +0000 UTC m=+2104.131736914" Nov 02 23:23:38.004885 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:38.004829 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-5674cb547f-cgtjh" podStartSLOduration=3.004813019 podStartE2EDuration="3.004813019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:23:38.003806317 +0000 UTC m=+2104.242303996" watchObservedRunningTime="2026-11-02 23:23:38.004813019 +0000 UTC m=+2104.243310681" Nov 02 23:23:41.242754 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:41.242720 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf"] Nov 02 23:23:41.247101 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:41.247082 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf" Nov 02 23:23:41.348467 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:41.348437 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e875bc-serving-cert\"" Nov 02 23:23:41.353432 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:41.353400 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf"] Nov 02 23:23:41.372189 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:41.372153 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9da6b802-026a-4118-9833-4802930688d8-maas-api-tls\") pod \"maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf\" (UID: \"9da6b802-026a-4118-9833-4802930688d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf" Nov 02 23:23:41.372316 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:41.372292 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da6b802-026a-4118-9833-4802930688d8-openshift-service-ca\") pod \"maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf\" (UID: \"9da6b802-026a-4118-9833-4802930688d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf" Nov 02 23:23:41.372383 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:41.372367 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9da6b802-026a-4118-9833-4802930688d8-metrics-certs\") pod \"maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf\" (UID: \"9da6b802-026a-4118-9833-4802930688d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf" Nov 02 23:23:41.372464 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:41.372451 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24ksk\" (UniqueName: \"kubernetes.io/projected/9da6b802-026a-4118-9833-4802930688d8-kube-api-access-24ksk\") pod \"maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf\" (UID: \"9da6b802-026a-4118-9833-4802930688d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf" Nov 02 23:23:41.473631 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:41.473593 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da6b802-026a-4118-9833-4802930688d8-openshift-service-ca\") pod \"maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf\" (UID: \"9da6b802-026a-4118-9833-4802930688d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf" Nov 02 23:23:41.473817 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:41.473668 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9da6b802-026a-4118-9833-4802930688d8-metrics-certs\") pod \"maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf\" (UID: \"9da6b802-026a-4118-9833-4802930688d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf" Nov 02 23:23:41.473817 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:41.473746 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24ksk\" (UniqueName: \"kubernetes.io/projected/9da6b802-026a-4118-9833-4802930688d8-kube-api-access-24ksk\") pod \"maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf\" (UID: \"9da6b802-026a-4118-9833-4802930688d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf" Nov 02 23:23:41.473921 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:41.473903 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9da6b802-026a-4118-9833-4802930688d8-maas-api-tls\") pod \"maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf\" (UID: \"9da6b802-026a-4118-9833-4802930688d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf" Nov 02 23:23:41.474061 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:23:41.474034 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-e875bc-serving-cert: secret "maas-api-e2e-api-b-e875bc-serving-cert" not found Nov 02 23:23:41.474179 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:23:41.474118 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9da6b802-026a-4118-9833-4802930688d8-maas-api-tls podName:9da6b802-026a-4118-9833-4802930688d8 nodeName:}" failed. No retries permitted until 2026-11-02 23:23:41.974098395 +0000 UTC m=+2108.212596048 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9da6b802-026a-4118-9833-4802930688d8-maas-api-tls") pod "maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf" (UID: "9da6b802-026a-4118-9833-4802930688d8") : secret "maas-api-e2e-api-b-e875bc-serving-cert" not found Nov 02 23:23:41.474402 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:41.474384 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da6b802-026a-4118-9833-4802930688d8-openshift-service-ca\") pod \"maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf\" (UID: \"9da6b802-026a-4118-9833-4802930688d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf" Nov 02 23:23:41.476799 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:41.476780 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9da6b802-026a-4118-9833-4802930688d8-metrics-certs\") pod \"maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf\" (UID: \"9da6b802-026a-4118-9833-4802930688d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf" Nov 02 23:23:41.536892 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:41.536812 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24ksk\" (UniqueName: \"kubernetes.io/projected/9da6b802-026a-4118-9833-4802930688d8-kube-api-access-24ksk\") pod \"maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf\" (UID: \"9da6b802-026a-4118-9833-4802930688d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf" Nov 02 23:23:41.980270 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:41.980230 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9da6b802-026a-4118-9833-4802930688d8-maas-api-tls\") pod \"maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf\" (UID: \"9da6b802-026a-4118-9833-4802930688d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf" Nov 02 23:23:41.983146 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:41.983093 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9da6b802-026a-4118-9833-4802930688d8-maas-api-tls\") pod \"maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf\" (UID: \"9da6b802-026a-4118-9833-4802930688d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf" Nov 02 23:23:42.164969 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:42.164933 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf" Nov 02 23:23:42.550262 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:42.550234 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf"] Nov 02 23:23:42.552010 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:23:42.551982 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9da6b802_026a_4118_9833_4802930688d8.slice/crio-5475ee1677b49ad6d5e189097952a6883c4b7237643726e32bc013f4d1a90ad4 WatchSource:0}: Error finding container 5475ee1677b49ad6d5e189097952a6883c4b7237643726e32bc013f4d1a90ad4: Status 404 returned error can't find the container with id 5475ee1677b49ad6d5e189097952a6883c4b7237643726e32bc013f4d1a90ad4 Nov 02 23:23:42.553646 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:42.553616 5727 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 23:23:42.815880 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:42.815846 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf" event={"ID":"9da6b802-026a-4118-9833-4802930688d8","Type":"ContainerStarted","Data":"5475ee1677b49ad6d5e189097952a6883c4b7237643726e32bc013f4d1a90ad4"} Nov 02 23:23:43.827974 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:43.827938 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf" event={"ID":"9da6b802-026a-4118-9833-4802930688d8","Type":"ContainerStarted","Data":"fb20c56d24a7d0f90440abcdf9ce95c6e9f3e0f7b3dc4ccab779fcc153276338"} Nov 02 23:23:43.828484 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:43.828013 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf" Nov 02 23:23:43.945589 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:43.945528 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf" podStartSLOduration=2.0801959 podStartE2EDuration="2.945510408s" totalImagesPullingTime="865.314508ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:23:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:23:43.945116593 +0000 UTC m=+2110.183614255" watchObservedRunningTime="2026-11-02 23:23:43.945510408 +0000 UTC m=+2110.184008071" Nov 02 23:23:48.786259 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:48.786231 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7549c7cff-j4lm4" Nov 02 23:23:48.786656 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:48.786309 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-5674cb547f-cgtjh" Nov 02 23:23:49.844999 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:23:49.844964 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf" Nov 02 23:23:57.845238 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:23:57.845202 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:24:03.050596 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:03.050561 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst"] Nov 02 23:24:03.054756 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:03.054727 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst" Nov 02 23:24:03.109303 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:03.109277 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-03fe6e-serving-cert\"" Nov 02 23:24:03.153879 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:03.153846 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33ee4560-1dc9-4b6c-a3b9-7376b13f5809-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst\" (UID: \"33ee4560-1dc9-4b6c-a3b9-7376b13f5809\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst" Nov 02 23:24:03.154076 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:03.153892 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33ee4560-1dc9-4b6c-a3b9-7376b13f5809-metrics-certs\") pod \"maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst\" (UID: \"33ee4560-1dc9-4b6c-a3b9-7376b13f5809\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst" Nov 02 23:24:03.154177 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:03.154160 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxfkl\" (UniqueName: \"kubernetes.io/projected/33ee4560-1dc9-4b6c-a3b9-7376b13f5809-kube-api-access-fxfkl\") pod \"maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst\" (UID: \"33ee4560-1dc9-4b6c-a3b9-7376b13f5809\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst" Nov 02 23:24:03.154218 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:03.154191 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33ee4560-1dc9-4b6c-a3b9-7376b13f5809-maas-api-tls\") pod \"maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst\" (UID: \"33ee4560-1dc9-4b6c-a3b9-7376b13f5809\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst" Nov 02 23:24:03.159773 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:03.159746 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst"] Nov 02 23:24:03.254825 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:03.254792 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxfkl\" (UniqueName: \"kubernetes.io/projected/33ee4560-1dc9-4b6c-a3b9-7376b13f5809-kube-api-access-fxfkl\") pod \"maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst\" (UID: \"33ee4560-1dc9-4b6c-a3b9-7376b13f5809\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst" Nov 02 23:24:03.254976 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:03.254828 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33ee4560-1dc9-4b6c-a3b9-7376b13f5809-maas-api-tls\") pod \"maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst\" (UID: \"33ee4560-1dc9-4b6c-a3b9-7376b13f5809\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst" Nov 02 23:24:03.254976 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:03.254869 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33ee4560-1dc9-4b6c-a3b9-7376b13f5809-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst\" (UID: \"33ee4560-1dc9-4b6c-a3b9-7376b13f5809\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst" Nov 02 23:24:03.254976 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:03.254910 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33ee4560-1dc9-4b6c-a3b9-7376b13f5809-metrics-certs\") pod \"maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst\" (UID: \"33ee4560-1dc9-4b6c-a3b9-7376b13f5809\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst" Nov 02 23:24:03.255094 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:24:03.255012 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-03fe6e-serving-cert: secret "maas-api-e2e-praxis-a-03fe6e-serving-cert" not found Nov 02 23:24:03.255094 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:24:03.255087 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33ee4560-1dc9-4b6c-a3b9-7376b13f5809-maas-api-tls podName:33ee4560-1dc9-4b6c-a3b9-7376b13f5809 nodeName:}" failed. No retries permitted until 2026-11-02 23:24:03.755068581 +0000 UTC m=+2129.993566222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/33ee4560-1dc9-4b6c-a3b9-7376b13f5809-maas-api-tls") pod "maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst" (UID: "33ee4560-1dc9-4b6c-a3b9-7376b13f5809") : secret "maas-api-e2e-praxis-a-03fe6e-serving-cert" not found Nov 02 23:24:03.255616 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:03.255591 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33ee4560-1dc9-4b6c-a3b9-7376b13f5809-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst\" (UID: \"33ee4560-1dc9-4b6c-a3b9-7376b13f5809\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst" Nov 02 23:24:03.257595 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:03.257574 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33ee4560-1dc9-4b6c-a3b9-7376b13f5809-metrics-certs\") pod \"maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst\" (UID: \"33ee4560-1dc9-4b6c-a3b9-7376b13f5809\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst" Nov 02 23:24:03.315874 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:03.315848 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxfkl\" (UniqueName: \"kubernetes.io/projected/33ee4560-1dc9-4b6c-a3b9-7376b13f5809-kube-api-access-fxfkl\") pod \"maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst\" (UID: \"33ee4560-1dc9-4b6c-a3b9-7376b13f5809\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst" Nov 02 23:24:03.760258 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:03.760169 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33ee4560-1dc9-4b6c-a3b9-7376b13f5809-maas-api-tls\") pod \"maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst\" (UID: \"33ee4560-1dc9-4b6c-a3b9-7376b13f5809\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst" Nov 02 23:24:03.762777 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:03.762748 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33ee4560-1dc9-4b6c-a3b9-7376b13f5809-maas-api-tls\") pod \"maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst\" (UID: \"33ee4560-1dc9-4b6c-a3b9-7376b13f5809\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst" Nov 02 23:24:03.968560 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:03.968523 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst" Nov 02 23:24:04.350562 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:04.350533 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst"] Nov 02 23:24:04.352460 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:24:04.352427 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ee4560_1dc9_4b6c_a3b9_7376b13f5809.slice/crio-8c502906304d5e826de5b038323f172c239dd4c5bb427f99bb7516320bf4b1e3 WatchSource:0}: Error finding container 8c502906304d5e826de5b038323f172c239dd4c5bb427f99bb7516320bf4b1e3: Status 404 returned error can't find the container with id 8c502906304d5e826de5b038323f172c239dd4c5bb427f99bb7516320bf4b1e3 Nov 02 23:24:05.006552 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:05.006518 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst" event={"ID":"33ee4560-1dc9-4b6c-a3b9-7376b13f5809","Type":"ContainerStarted","Data":"8c502906304d5e826de5b038323f172c239dd4c5bb427f99bb7516320bf4b1e3"} Nov 02 23:24:06.016149 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:06.016096 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst" event={"ID":"33ee4560-1dc9-4b6c-a3b9-7376b13f5809","Type":"ContainerStarted","Data":"4be4a7cd10272fdfabb6d0b4a6767cf78c01fdd8455d081f01d440f6ad2322ba"} Nov 02 23:24:06.016557 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:06.016202 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst" Nov 02 23:24:06.136195 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:06.136142 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst" podStartSLOduration=2.3288658939999998 podStartE2EDuration="3.13610856s" totalImagesPullingTime="807.242666ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:24:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:24:06.13427406 +0000 UTC m=+2132.372771749" watchObservedRunningTime="2026-11-02 23:24:06.13610856 +0000 UTC m=+2132.374606247" Nov 02 23:24:12.036419 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:12.036390 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst" Nov 02 23:24:27.846420 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:24:27.846372 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:24:44.993862 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:44.993774 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp"] Nov 02 23:24:45.000449 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:45.000422 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp" Nov 02 23:24:45.053746 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:45.053714 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp"] Nov 02 23:24:45.100015 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:45.099338 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60f51458-ff9e-4b5d-97b1-f9f4ef10f580-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp\" (UID: \"60f51458-ff9e-4b5d-97b1-f9f4ef10f580\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp" Nov 02 23:24:45.100015 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:45.099401 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60f51458-ff9e-4b5d-97b1-f9f4ef10f580-metrics-certs\") pod \"maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp\" (UID: \"60f51458-ff9e-4b5d-97b1-f9f4ef10f580\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp" Nov 02 23:24:45.100015 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:45.099423 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-014ab4-serving-cert\"" Nov 02 23:24:45.100015 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:45.099558 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60f51458-ff9e-4b5d-97b1-f9f4ef10f580-maas-api-tls\") pod \"maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp\" (UID: \"60f51458-ff9e-4b5d-97b1-f9f4ef10f580\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp" Nov 02 23:24:45.100015 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:45.099628 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwvzl\" (UniqueName: \"kubernetes.io/projected/60f51458-ff9e-4b5d-97b1-f9f4ef10f580-kube-api-access-fwvzl\") pod \"maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp\" (UID: \"60f51458-ff9e-4b5d-97b1-f9f4ef10f580\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp" Nov 02 23:24:45.200530 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:45.200490 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60f51458-ff9e-4b5d-97b1-f9f4ef10f580-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp\" (UID: \"60f51458-ff9e-4b5d-97b1-f9f4ef10f580\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp" Nov 02 23:24:45.200755 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:45.200542 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60f51458-ff9e-4b5d-97b1-f9f4ef10f580-metrics-certs\") pod \"maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp\" (UID: \"60f51458-ff9e-4b5d-97b1-f9f4ef10f580\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp" Nov 02 23:24:45.200755 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:45.200666 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60f51458-ff9e-4b5d-97b1-f9f4ef10f580-maas-api-tls\") pod \"maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp\" (UID: \"60f51458-ff9e-4b5d-97b1-f9f4ef10f580\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp" Nov 02 23:24:45.200755 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:45.200721 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwvzl\" (UniqueName: \"kubernetes.io/projected/60f51458-ff9e-4b5d-97b1-f9f4ef10f580-kube-api-access-fwvzl\") pod \"maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp\" (UID: \"60f51458-ff9e-4b5d-97b1-f9f4ef10f580\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp" Nov 02 23:24:45.200930 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:24:45.200816 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-014ab4-serving-cert: secret "maas-api-e2e-ipp-b-014ab4-serving-cert" not found Nov 02 23:24:45.200930 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:24:45.200898 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60f51458-ff9e-4b5d-97b1-f9f4ef10f580-maas-api-tls podName:60f51458-ff9e-4b5d-97b1-f9f4ef10f580 nodeName:}" failed. No retries permitted until 2026-11-02 23:24:45.700880526 +0000 UTC m=+2171.939378170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/60f51458-ff9e-4b5d-97b1-f9f4ef10f580-maas-api-tls") pod "maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp" (UID: "60f51458-ff9e-4b5d-97b1-f9f4ef10f580") : secret "maas-api-e2e-ipp-b-014ab4-serving-cert" not found Nov 02 23:24:45.201262 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:45.201241 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60f51458-ff9e-4b5d-97b1-f9f4ef10f580-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp\" (UID: \"60f51458-ff9e-4b5d-97b1-f9f4ef10f580\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp" Nov 02 23:24:45.203528 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:45.203508 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60f51458-ff9e-4b5d-97b1-f9f4ef10f580-metrics-certs\") pod \"maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp\" (UID: \"60f51458-ff9e-4b5d-97b1-f9f4ef10f580\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp" Nov 02 23:24:45.264315 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:45.264180 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwvzl\" (UniqueName: \"kubernetes.io/projected/60f51458-ff9e-4b5d-97b1-f9f4ef10f580-kube-api-access-fwvzl\") pod \"maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp\" (UID: \"60f51458-ff9e-4b5d-97b1-f9f4ef10f580\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp" Nov 02 23:24:45.705968 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:45.705921 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60f51458-ff9e-4b5d-97b1-f9f4ef10f580-maas-api-tls\") pod \"maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp\" (UID: \"60f51458-ff9e-4b5d-97b1-f9f4ef10f580\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp" Nov 02 23:24:45.708638 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:45.708615 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60f51458-ff9e-4b5d-97b1-f9f4ef10f580-maas-api-tls\") pod \"maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp\" (UID: \"60f51458-ff9e-4b5d-97b1-f9f4ef10f580\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp" Nov 02 23:24:45.915030 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:45.915000 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp" Nov 02 23:24:46.300277 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:46.300249 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp"] Nov 02 23:24:46.302073 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:24:46.302045 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f51458_ff9e_4b5d_97b1_f9f4ef10f580.slice/crio-9839432a391831ad47ac51a3b134bbd1e888a63f8bf8210242fa0dbe5b106721 WatchSource:0}: Error finding container 9839432a391831ad47ac51a3b134bbd1e888a63f8bf8210242fa0dbe5b106721: Status 404 returned error can't find the container with id 9839432a391831ad47ac51a3b134bbd1e888a63f8bf8210242fa0dbe5b106721 Nov 02 23:24:46.343599 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:46.343566 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp" event={"ID":"60f51458-ff9e-4b5d-97b1-f9f4ef10f580","Type":"ContainerStarted","Data":"9839432a391831ad47ac51a3b134bbd1e888a63f8bf8210242fa0dbe5b106721"} Nov 02 23:24:47.355019 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:47.354976 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp" event={"ID":"60f51458-ff9e-4b5d-97b1-f9f4ef10f580","Type":"ContainerStarted","Data":"6bbb47ab9c9901d07f8af644861012b1d52312d3154cea2a7ad31bbd52be90e7"} Nov 02 23:24:47.355473 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:47.355047 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp" Nov 02 23:24:47.474184 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:47.474079 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp" podStartSLOduration=2.628555997 podStartE2EDuration="3.474064016s" totalImagesPullingTime="845.508019ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:24:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:24:47.472040839 +0000 UTC m=+2173.710538502" watchObservedRunningTime="2026-11-02 23:24:47.474064016 +0000 UTC m=+2173.712561678" Nov 02 23:24:53.370548 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:53.370505 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp" Nov 02 23:24:55.486417 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:55.486382 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm"] Nov 02 23:24:55.486870 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:55.486623 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" podUID="40350d65-8ea1-4d78-882c-f7b75c61dccd" containerName="istio-proxy" containerID="cri-o://b283b41806dadd842746ea010d794d131af68310ac330487f0ae237624d9eaf1" gracePeriod=30 Nov 02 23:24:56.205947 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:56.205910 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs"] Nov 02 23:24:56.206146 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:56.206109 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" podUID="bfdaf98c-fc85-4457-8366-4a81aab5573a" containerName="istio-proxy" containerID="cri-o://d5286ad12669a017e9a8891815dcbe109d13f8e1a6f3659f4649b36298c0fe27" gracePeriod=30 Nov 02 23:24:57.846894 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:24:57.846856 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:24:58.020823 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:58.020790 5727 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:24:58.020985 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:58.020856 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" podUID="bfdaf98c-fc85-4457-8366-4a81aab5573a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:24:59.922662 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:59.922625 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:24:59.923033 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:24:59.922687 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" podUID="40350d65-8ea1-4d78-882c-f7b75c61dccd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:25:00.764379 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:00.764310 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:25:00.837620 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:00.837584 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40350d65-8ea1-4d78-882c-f7b75c61dccd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40350d65-8ea1-4d78-882c-f7b75c61dccd-istio-podinfo\") pod \"40350d65-8ea1-4d78-882c-f7b75c61dccd\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " Nov 02 23:25:00.837781 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:00.837630 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-credential-socket\") pod \"40350d65-8ea1-4d78-882c-f7b75c61dccd\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " Nov 02 23:25:00.837781 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:00.837656 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-workload-certs\") pod \"40350d65-8ea1-4d78-882c-f7b75c61dccd\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " Nov 02 23:25:00.837781 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:00.837691 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-istio-data\") pod \"40350d65-8ea1-4d78-882c-f7b75c61dccd\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " Nov 02 23:25:00.837951 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:00.837777 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40350d65-8ea1-4d78-882c-f7b75c61dccd-kube-api-access-2nsbr\" (UniqueName: \"kubernetes.io/projected/40350d65-8ea1-4d78-882c-f7b75c61dccd-kube-api-access-2nsbr\") pod \"40350d65-8ea1-4d78-882c-f7b75c61dccd\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " Nov 02 23:25:00.837951 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:00.837844 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40350d65-8ea1-4d78-882c-f7b75c61dccd-istio-token\" (UniqueName: \"kubernetes.io/projected/40350d65-8ea1-4d78-882c-f7b75c61dccd-istio-token\") pod \"40350d65-8ea1-4d78-882c-f7b75c61dccd\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " Nov 02 23:25:00.838050 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:00.837942 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-workload-certs" pod "40350d65-8ea1-4d78-882c-f7b75c61dccd" (UID: "40350d65-8ea1-4d78-882c-f7b75c61dccd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:00.838050 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:00.837959 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-credential-socket" pod "40350d65-8ea1-4d78-882c-f7b75c61dccd" (UID: "40350d65-8ea1-4d78-882c-f7b75c61dccd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:00.838050 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:00.838024 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-istio-data" pod "40350d65-8ea1-4d78-882c-f7b75c61dccd" (UID: "40350d65-8ea1-4d78-882c-f7b75c61dccd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:00.838238 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:00.838047 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-workload-socket\") pod \"40350d65-8ea1-4d78-882c-f7b75c61dccd\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " Nov 02 23:25:00.838238 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:00.838109 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40350d65-8ea1-4d78-882c-f7b75c61dccd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40350d65-8ea1-4d78-882c-f7b75c61dccd-istiod-ca-cert\") pod \"40350d65-8ea1-4d78-882c-f7b75c61dccd\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " Nov 02 23:25:00.838238 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:00.838191 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-istio-envoy\") pod \"40350d65-8ea1-4d78-882c-f7b75c61dccd\" (UID: \"40350d65-8ea1-4d78-882c-f7b75c61dccd\") " Nov 02 23:25:00.838401 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:00.838307 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-workload-socket" pod "40350d65-8ea1-4d78-882c-f7b75c61dccd" (UID: "40350d65-8ea1-4d78-882c-f7b75c61dccd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:00.838598 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:00.838535 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40350d65-8ea1-4d78-882c-f7b75c61dccd-istiod-ca-cert" pod "40350d65-8ea1-4d78-882c-f7b75c61dccd" (UID: "40350d65-8ea1-4d78-882c-f7b75c61dccd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:25:00.838827 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:00.838801 5727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-workload-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:00.838899 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:00.838832 5727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40350d65-8ea1-4d78-882c-f7b75c61dccd-istiod-ca-cert\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:00.838899 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:00.838848 5727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-credential-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:00.839020 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:00.838864 5727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-workload-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:00.839020 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:00.838968 5727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-istio-data\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:00.840333 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:00.840310 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40350d65-8ea1-4d78-882c-f7b75c61dccd-kube-api-access-2nsbr" pod "40350d65-8ea1-4d78-882c-f7b75c61dccd" (UID: "40350d65-8ea1-4d78-882c-f7b75c61dccd"). InnerVolumeSpecName "kube-api-access-2nsbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:25:00.840658 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:00.840623 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-istio-envoy" pod "40350d65-8ea1-4d78-882c-f7b75c61dccd" (UID: "40350d65-8ea1-4d78-882c-f7b75c61dccd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:00.840768 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:00.840709 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40350d65-8ea1-4d78-882c-f7b75c61dccd-istio-podinfo" pod "40350d65-8ea1-4d78-882c-f7b75c61dccd" (UID: "40350d65-8ea1-4d78-882c-f7b75c61dccd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:25:00.840832 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:00.840783 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40350d65-8ea1-4d78-882c-f7b75c61dccd-istio-token" pod "40350d65-8ea1-4d78-882c-f7b75c61dccd" (UID: "40350d65-8ea1-4d78-882c-f7b75c61dccd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:25:00.939899 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:00.939864 5727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40350d65-8ea1-4d78-882c-f7b75c61dccd-istio-token\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:00.939899 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:00.939898 5727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40350d65-8ea1-4d78-882c-f7b75c61dccd-istio-envoy\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:00.939899 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:00.939913 5727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40350d65-8ea1-4d78-882c-f7b75c61dccd-istio-podinfo\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:00.940415 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:00.939930 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nsbr\" (UniqueName: \"kubernetes.io/projected/40350d65-8ea1-4d78-882c-f7b75c61dccd-kube-api-access-2nsbr\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:01.296368 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.296337 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv"] Nov 02 23:25:01.296647 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.296607 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" podUID="3bd21a3d-ad0a-48db-80fa-37b40e05bb74" containerName="istio-proxy" containerID="cri-o://9009a2c0770b86b983910d054034e182e5ee9b02eec7b2b7fe2c7894f7ba53f8" gracePeriod=30 Nov 02 23:25:01.464673 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.464642 5727 generic.go:356] "Generic (PLEG): container finished" podID="bfdaf98c-fc85-4457-8366-4a81aab5573a" containerID="d5286ad12669a017e9a8891815dcbe109d13f8e1a6f3659f4649b36298c0fe27" exitCode=0 Nov 02 23:25:01.464845 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.464734 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" event={"ID":"bfdaf98c-fc85-4457-8366-4a81aab5573a","Type":"ContainerDied","Data":"d5286ad12669a017e9a8891815dcbe109d13f8e1a6f3659f4649b36298c0fe27"} Nov 02 23:25:01.466368 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.466337 5727 generic.go:356] "Generic (PLEG): container finished" podID="40350d65-8ea1-4d78-882c-f7b75c61dccd" containerID="b283b41806dadd842746ea010d794d131af68310ac330487f0ae237624d9eaf1" exitCode=0 Nov 02 23:25:01.466509 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.466398 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" Nov 02 23:25:01.466509 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.466426 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" event={"ID":"40350d65-8ea1-4d78-882c-f7b75c61dccd","Type":"ContainerDied","Data":"b283b41806dadd842746ea010d794d131af68310ac330487f0ae237624d9eaf1"} Nov 02 23:25:01.466509 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.466480 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm" event={"ID":"40350d65-8ea1-4d78-882c-f7b75c61dccd","Type":"ContainerDied","Data":"98bed883dde4e655b622ec3bfdafb3916b1687aa661f9591ca865e813877afc4"} Nov 02 23:25:01.466678 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.466520 5727 scope.go:122] "RemoveContainer" containerID="b283b41806dadd842746ea010d794d131af68310ac330487f0ae237624d9eaf1" Nov 02 23:25:01.480000 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.479983 5727 scope.go:122] "RemoveContainer" containerID="b283b41806dadd842746ea010d794d131af68310ac330487f0ae237624d9eaf1" Nov 02 23:25:01.480351 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:25:01.480322 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b283b41806dadd842746ea010d794d131af68310ac330487f0ae237624d9eaf1\": container with ID starting with b283b41806dadd842746ea010d794d131af68310ac330487f0ae237624d9eaf1 not found: ID does not exist" containerID="b283b41806dadd842746ea010d794d131af68310ac330487f0ae237624d9eaf1" Nov 02 23:25:01.480427 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.480365 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b283b41806dadd842746ea010d794d131af68310ac330487f0ae237624d9eaf1"} err="failed to get container status \"b283b41806dadd842746ea010d794d131af68310ac330487f0ae237624d9eaf1\": rpc error: code = NotFound desc = could not find container \"b283b41806dadd842746ea010d794d131af68310ac330487f0ae237624d9eaf1\": container with ID starting with b283b41806dadd842746ea010d794d131af68310ac330487f0ae237624d9eaf1 not found: ID does not exist" Nov 02 23:25:01.490831 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.490810 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:25:01.547025 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.547001 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-workload-certs\") pod \"bfdaf98c-fc85-4457-8366-4a81aab5573a\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " Nov 02 23:25:01.547172 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.547062 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-credential-socket\") pod \"bfdaf98c-fc85-4457-8366-4a81aab5573a\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " Nov 02 23:25:01.547172 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.547100 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-workload-socket\") pod \"bfdaf98c-fc85-4457-8366-4a81aab5573a\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " Nov 02 23:25:01.547299 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.547211 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfdaf98c-fc85-4457-8366-4a81aab5573a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfdaf98c-fc85-4457-8366-4a81aab5573a-istio-podinfo\") pod \"bfdaf98c-fc85-4457-8366-4a81aab5573a\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " Nov 02 23:25:01.547299 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.547274 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfdaf98c-fc85-4457-8366-4a81aab5573a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfdaf98c-fc85-4457-8366-4a81aab5573a-istiod-ca-cert\") pod \"bfdaf98c-fc85-4457-8366-4a81aab5573a\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " Nov 02 23:25:01.547405 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.547359 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-credential-socket" pod "bfdaf98c-fc85-4457-8366-4a81aab5573a" (UID: "bfdaf98c-fc85-4457-8366-4a81aab5573a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:01.547405 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.547373 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-istio-data\") pod \"bfdaf98c-fc85-4457-8366-4a81aab5573a\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " Nov 02 23:25:01.547512 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.547475 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-workload-certs" pod "bfdaf98c-fc85-4457-8366-4a81aab5573a" (UID: "bfdaf98c-fc85-4457-8366-4a81aab5573a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:01.547512 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.547483 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-workload-socket" pod "bfdaf98c-fc85-4457-8366-4a81aab5573a" (UID: "bfdaf98c-fc85-4457-8366-4a81aab5573a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:01.547512 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.547486 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-istio-envoy\") pod \"bfdaf98c-fc85-4457-8366-4a81aab5573a\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " Nov 02 23:25:01.547669 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.547628 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfdaf98c-fc85-4457-8366-4a81aab5573a-kube-api-access-plmh5\" (UniqueName: \"kubernetes.io/projected/bfdaf98c-fc85-4457-8366-4a81aab5573a-kube-api-access-plmh5\") pod \"bfdaf98c-fc85-4457-8366-4a81aab5573a\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " Nov 02 23:25:01.547723 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.547694 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-istio-data" pod "bfdaf98c-fc85-4457-8366-4a81aab5573a" (UID: "bfdaf98c-fc85-4457-8366-4a81aab5573a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:01.547723 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.547700 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfdaf98c-fc85-4457-8366-4a81aab5573a-istiod-ca-cert" pod "bfdaf98c-fc85-4457-8366-4a81aab5573a" (UID: "bfdaf98c-fc85-4457-8366-4a81aab5573a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:25:01.548325 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.548301 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfdaf98c-fc85-4457-8366-4a81aab5573a-istio-token\" (UniqueName: \"kubernetes.io/projected/bfdaf98c-fc85-4457-8366-4a81aab5573a-istio-token\") pod \"bfdaf98c-fc85-4457-8366-4a81aab5573a\" (UID: \"bfdaf98c-fc85-4457-8366-4a81aab5573a\") " Nov 02 23:25:01.548979 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.548952 5727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-workload-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:01.549108 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.548984 5727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-credential-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:01.549108 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.549000 5727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-workload-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:01.549108 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.549013 5727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfdaf98c-fc85-4457-8366-4a81aab5573a-istiod-ca-cert\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:01.549108 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.549027 5727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-istio-data\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:01.549985 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.549959 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfdaf98c-fc85-4457-8366-4a81aab5573a-kube-api-access-plmh5" pod "bfdaf98c-fc85-4457-8366-4a81aab5573a" (UID: "bfdaf98c-fc85-4457-8366-4a81aab5573a"). InnerVolumeSpecName "kube-api-access-plmh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:25:01.550085 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.550011 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfdaf98c-fc85-4457-8366-4a81aab5573a-istio-podinfo" pod "bfdaf98c-fc85-4457-8366-4a81aab5573a" (UID: "bfdaf98c-fc85-4457-8366-4a81aab5573a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:25:01.551331 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.551300 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfdaf98c-fc85-4457-8366-4a81aab5573a-istio-token" pod "bfdaf98c-fc85-4457-8366-4a81aab5573a" (UID: "bfdaf98c-fc85-4457-8366-4a81aab5573a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:25:01.552512 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.552489 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-istio-envoy" pod "bfdaf98c-fc85-4457-8366-4a81aab5573a" (UID: "bfdaf98c-fc85-4457-8366-4a81aab5573a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:01.652714 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.649946 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plmh5\" (UniqueName: \"kubernetes.io/projected/bfdaf98c-fc85-4457-8366-4a81aab5573a-kube-api-access-plmh5\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:01.652714 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.649982 5727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfdaf98c-fc85-4457-8366-4a81aab5573a-istio-token\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:01.652714 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.649996 5727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfdaf98c-fc85-4457-8366-4a81aab5573a-istio-podinfo\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:01.652714 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.650012 5727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfdaf98c-fc85-4457-8366-4a81aab5573a-istio-envoy\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:01.672607 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.672578 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9"] Nov 02 23:25:01.672834 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.672793 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9" podUID="96c11959-99bc-467a-a844-74fc983c126e" containerName="maas-api" containerID="cri-o://a087d729445696926f6289cb9af47839f9240032608112928be53f3b8e5c6596" gracePeriod=30 Nov 02 23:25:01.755286 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.755097 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm"] Nov 02 23:25:01.758702 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.758673 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-9khcm"] Nov 02 23:25:01.905284 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.905249 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40350d65-8ea1-4d78-882c-f7b75c61dccd" path="/var/lib/kubelet/pods/40350d65-8ea1-4d78-882c-f7b75c61dccd/volumes" Nov 02 23:25:01.953996 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:01.953971 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9" Nov 02 23:25:02.057314 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:02.057234 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96c11959-99bc-467a-a844-74fc983c126e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/96c11959-99bc-467a-a844-74fc983c126e-metrics-certs\") pod \"96c11959-99bc-467a-a844-74fc983c126e\" (UID: \"96c11959-99bc-467a-a844-74fc983c126e\") " Nov 02 23:25:02.057488 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:02.057467 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96c11959-99bc-467a-a844-74fc983c126e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96c11959-99bc-467a-a844-74fc983c126e-maas-api-tls\") pod \"96c11959-99bc-467a-a844-74fc983c126e\" (UID: \"96c11959-99bc-467a-a844-74fc983c126e\") " Nov 02 23:25:02.057562 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:02.057519 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96c11959-99bc-467a-a844-74fc983c126e-kube-api-access-rhnql\" (UniqueName: \"kubernetes.io/projected/96c11959-99bc-467a-a844-74fc983c126e-kube-api-access-rhnql\") pod \"96c11959-99bc-467a-a844-74fc983c126e\" (UID: \"96c11959-99bc-467a-a844-74fc983c126e\") " Nov 02 23:25:02.057631 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:02.057614 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96c11959-99bc-467a-a844-74fc983c126e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96c11959-99bc-467a-a844-74fc983c126e-openshift-service-ca\") pod \"96c11959-99bc-467a-a844-74fc983c126e\" (UID: \"96c11959-99bc-467a-a844-74fc983c126e\") " Nov 02 23:25:02.072152 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:02.070405 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96c11959-99bc-467a-a844-74fc983c126e-openshift-service-ca" pod "96c11959-99bc-467a-a844-74fc983c126e" (UID: "96c11959-99bc-467a-a844-74fc983c126e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:25:02.076812 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:02.076748 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96c11959-99bc-467a-a844-74fc983c126e-kube-api-access-rhnql" pod "96c11959-99bc-467a-a844-74fc983c126e" (UID: "96c11959-99bc-467a-a844-74fc983c126e"). InnerVolumeSpecName "kube-api-access-rhnql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:25:02.102714 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:02.102671 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96c11959-99bc-467a-a844-74fc983c126e-maas-api-tls" pod "96c11959-99bc-467a-a844-74fc983c126e" (UID: "96c11959-99bc-467a-a844-74fc983c126e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:25:02.105257 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:02.105225 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96c11959-99bc-467a-a844-74fc983c126e-metrics-certs" pod "96c11959-99bc-467a-a844-74fc983c126e" (UID: "96c11959-99bc-467a-a844-74fc983c126e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:25:02.170931 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:02.170896 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96c11959-99bc-467a-a844-74fc983c126e-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:02.170931 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:02.170923 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhnql\" (UniqueName: \"kubernetes.io/projected/96c11959-99bc-467a-a844-74fc983c126e-kube-api-access-rhnql\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:02.171118 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:02.170938 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96c11959-99bc-467a-a844-74fc983c126e-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:02.171118 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:02.170952 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96c11959-99bc-467a-a844-74fc983c126e-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:02.477585 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:02.477554 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" Nov 02 23:25:02.477585 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:02.477570 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs" event={"ID":"bfdaf98c-fc85-4457-8366-4a81aab5573a","Type":"ContainerDied","Data":"4d1b942326ddb82a0cc157deeda02a4a946b8d3fc9067c3ab54b10d0be3738af"} Nov 02 23:25:02.477850 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:02.477654 5727 scope.go:122] "RemoveContainer" containerID="d5286ad12669a017e9a8891815dcbe109d13f8e1a6f3659f4649b36298c0fe27" Nov 02 23:25:02.480729 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:02.480698 5727 generic.go:356] "Generic (PLEG): container finished" podID="96c11959-99bc-467a-a844-74fc983c126e" containerID="a087d729445696926f6289cb9af47839f9240032608112928be53f3b8e5c6596" exitCode=0 Nov 02 23:25:02.480729 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:02.480735 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9" Nov 02 23:25:02.480943 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:02.480779 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9" event={"ID":"96c11959-99bc-467a-a844-74fc983c126e","Type":"ContainerDied","Data":"a087d729445696926f6289cb9af47839f9240032608112928be53f3b8e5c6596"} Nov 02 23:25:02.480943 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:02.480843 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9" event={"ID":"96c11959-99bc-467a-a844-74fc983c126e","Type":"ContainerDied","Data":"f985ae02bf1d2443fd61a533aeb7abefdf0a382fb80846e77cc55f6d700611cc"} Nov 02 23:25:02.491021 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:02.491006 5727 scope.go:122] "RemoveContainer" containerID="a087d729445696926f6289cb9af47839f9240032608112928be53f3b8e5c6596" Nov 02 23:25:02.501511 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:02.501491 5727 scope.go:122] "RemoveContainer" containerID="a087d729445696926f6289cb9af47839f9240032608112928be53f3b8e5c6596" Nov 02 23:25:02.501833 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:25:02.501800 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a087d729445696926f6289cb9af47839f9240032608112928be53f3b8e5c6596\": container with ID starting with a087d729445696926f6289cb9af47839f9240032608112928be53f3b8e5c6596 not found: ID does not exist" containerID="a087d729445696926f6289cb9af47839f9240032608112928be53f3b8e5c6596" Nov 02 23:25:02.501904 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:02.501829 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a087d729445696926f6289cb9af47839f9240032608112928be53f3b8e5c6596"} err="failed to get container status \"a087d729445696926f6289cb9af47839f9240032608112928be53f3b8e5c6596\": rpc error: code = NotFound desc = could not find container \"a087d729445696926f6289cb9af47839f9240032608112928be53f3b8e5c6596\": container with ID starting with a087d729445696926f6289cb9af47839f9240032608112928be53f3b8e5c6596 not found: ID does not exist" Nov 02 23:25:02.679277 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:02.679239 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9"] Nov 02 23:25:02.686327 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:02.686268 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-92072f-895f65cfc-7gxp9"] Nov 02 23:25:02.779096 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:02.778974 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs"] Nov 02 23:25:02.784240 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:02.784211 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-2cxcs"] Nov 02 23:25:03.725679 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:03.725641 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df"] Nov 02 23:25:03.726174 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:03.725976 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" podUID="65ee46cd-bfd4-439b-a15c-394e7d10b915" containerName="istio-proxy" containerID="cri-o://b9a106765c32a5935e454d98227962009e2693dd6e666a0557e50e196c12c2b6" gracePeriod=30 Nov 02 23:25:03.885474 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:03.885433 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96c11959-99bc-467a-a844-74fc983c126e" path="/var/lib/kubelet/pods/96c11959-99bc-467a-a844-74fc983c126e/volumes" Nov 02 23:25:03.885935 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:03.885919 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfdaf98c-fc85-4457-8366-4a81aab5573a" path="/var/lib/kubelet/pods/bfdaf98c-fc85-4457-8366-4a81aab5573a/volumes" Nov 02 23:25:05.924621 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:05.924583 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-7g7df container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:25:05.925100 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:05.924658 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" podUID="65ee46cd-bfd4-439b-a15c-394e7d10b915" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:25:06.529217 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:06.529184 5727 generic.go:356] "Generic (PLEG): container finished" podID="3bd21a3d-ad0a-48db-80fa-37b40e05bb74" containerID="9009a2c0770b86b983910d054034e182e5ee9b02eec7b2b7fe2c7894f7ba53f8" exitCode=0 Nov 02 23:25:06.529393 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:06.529260 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" event={"ID":"3bd21a3d-ad0a-48db-80fa-37b40e05bb74","Type":"ContainerDied","Data":"9009a2c0770b86b983910d054034e182e5ee9b02eec7b2b7fe2c7894f7ba53f8"} Nov 02 23:25:06.593121 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:06.593095 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:25:06.743267 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:06.743239 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istio-data\") pod \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " Nov 02 23:25:06.743512 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:06.743279 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istio-token\" (UniqueName: \"kubernetes.io/projected/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istio-token\") pod \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " Nov 02 23:25:06.743512 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:06.743359 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istiod-ca-cert\") pod \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " Nov 02 23:25:06.743512 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:06.743416 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istio-envoy\") pod \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " Nov 02 23:25:06.743694 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:06.743518 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istio-podinfo\") pod \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " Nov 02 23:25:06.743694 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:06.743548 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-kube-api-access-gwl59\" (UniqueName: \"kubernetes.io/projected/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-kube-api-access-gwl59\") pod \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " Nov 02 23:25:06.743694 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:06.743591 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-workload-socket\") pod \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " Nov 02 23:25:06.743694 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:06.743596 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istio-data" pod "3bd21a3d-ad0a-48db-80fa-37b40e05bb74" (UID: "3bd21a3d-ad0a-48db-80fa-37b40e05bb74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:06.743694 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:06.743640 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-credential-socket\") pod \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " Nov 02 23:25:06.743962 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:06.743710 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-workload-certs\") pod \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\" (UID: \"3bd21a3d-ad0a-48db-80fa-37b40e05bb74\") " Nov 02 23:25:06.743962 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:06.743806 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istiod-ca-cert" pod "3bd21a3d-ad0a-48db-80fa-37b40e05bb74" (UID: "3bd21a3d-ad0a-48db-80fa-37b40e05bb74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:25:06.743962 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:06.743856 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-workload-socket" pod "3bd21a3d-ad0a-48db-80fa-37b40e05bb74" (UID: "3bd21a3d-ad0a-48db-80fa-37b40e05bb74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:06.744120 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:06.744071 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-workload-certs" pod "3bd21a3d-ad0a-48db-80fa-37b40e05bb74" (UID: "3bd21a3d-ad0a-48db-80fa-37b40e05bb74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:06.744120 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:06.744090 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-credential-socket" pod "3bd21a3d-ad0a-48db-80fa-37b40e05bb74" (UID: "3bd21a3d-ad0a-48db-80fa-37b40e05bb74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:06.744522 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:06.744500 5727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-workload-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:06.744620 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:06.744526 5727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-credential-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:06.744620 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:06.744542 5727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-workload-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:06.744620 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:06.744559 5727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istio-data\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:06.744620 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:06.744574 5727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istiod-ca-cert\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:06.745829 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:06.745802 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istio-podinfo" pod "3bd21a3d-ad0a-48db-80fa-37b40e05bb74" (UID: "3bd21a3d-ad0a-48db-80fa-37b40e05bb74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:25:06.745966 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:06.745939 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istio-token" pod "3bd21a3d-ad0a-48db-80fa-37b40e05bb74" (UID: "3bd21a3d-ad0a-48db-80fa-37b40e05bb74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:25:06.746348 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:06.746327 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istio-envoy" pod "3bd21a3d-ad0a-48db-80fa-37b40e05bb74" (UID: "3bd21a3d-ad0a-48db-80fa-37b40e05bb74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:06.746348 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:06.746333 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-kube-api-access-gwl59" pod "3bd21a3d-ad0a-48db-80fa-37b40e05bb74" (UID: "3bd21a3d-ad0a-48db-80fa-37b40e05bb74"). InnerVolumeSpecName "kube-api-access-gwl59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:25:06.846342 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:06.846267 5727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istio-envoy\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:06.846342 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:06.846294 5727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istio-podinfo\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:06.846342 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:06.846305 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwl59\" (UniqueName: \"kubernetes.io/projected/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-kube-api-access-gwl59\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:06.846342 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:06.846315 5727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bd21a3d-ad0a-48db-80fa-37b40e05bb74-istio-token\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:07.537201 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:07.537174 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" Nov 02 23:25:07.537201 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:07.537180 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv" event={"ID":"3bd21a3d-ad0a-48db-80fa-37b40e05bb74","Type":"ContainerDied","Data":"2a61943e6e8047a178d7092b02af939a6a995a661ee0d931cce471e9fd4681ce"} Nov 02 23:25:07.537694 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:07.537266 5727 scope.go:122] "RemoveContainer" containerID="9009a2c0770b86b983910d054034e182e5ee9b02eec7b2b7fe2c7894f7ba53f8" Nov 02 23:25:07.702487 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:07.702450 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv"] Nov 02 23:25:07.752754 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:07.752725 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8jrgv"] Nov 02 23:25:07.887310 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:07.887270 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bd21a3d-ad0a-48db-80fa-37b40e05bb74" path="/var/lib/kubelet/pods/3bd21a3d-ad0a-48db-80fa-37b40e05bb74/volumes" Nov 02 23:25:09.040168 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.040142 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:25:09.177074 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.177041 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-credential-socket\") pod \"65ee46cd-bfd4-439b-a15c-394e7d10b915\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " Nov 02 23:25:09.177260 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.177084 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65ee46cd-bfd4-439b-a15c-394e7d10b915-kube-api-access-nxdx8\" (UniqueName: \"kubernetes.io/projected/65ee46cd-bfd4-439b-a15c-394e7d10b915-kube-api-access-nxdx8\") pod \"65ee46cd-bfd4-439b-a15c-394e7d10b915\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " Nov 02 23:25:09.177260 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.177109 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-workload-socket\") pod \"65ee46cd-bfd4-439b-a15c-394e7d10b915\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " Nov 02 23:25:09.177260 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.177184 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-istio-data\") pod \"65ee46cd-bfd4-439b-a15c-394e7d10b915\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " Nov 02 23:25:09.177260 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.177205 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65ee46cd-bfd4-439b-a15c-394e7d10b915-istio-token\" (UniqueName: \"kubernetes.io/projected/65ee46cd-bfd4-439b-a15c-394e7d10b915-istio-token\") pod \"65ee46cd-bfd4-439b-a15c-394e7d10b915\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " Nov 02 23:25:09.177535 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.177292 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65ee46cd-bfd4-439b-a15c-394e7d10b915-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ee46cd-bfd4-439b-a15c-394e7d10b915-istiod-ca-cert\") pod \"65ee46cd-bfd4-439b-a15c-394e7d10b915\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " Nov 02 23:25:09.177535 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.177395 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65ee46cd-bfd4-439b-a15c-394e7d10b915-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ee46cd-bfd4-439b-a15c-394e7d10b915-istio-podinfo\") pod \"65ee46cd-bfd4-439b-a15c-394e7d10b915\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " Nov 02 23:25:09.177535 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.177461 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-workload-certs\") pod \"65ee46cd-bfd4-439b-a15c-394e7d10b915\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " Nov 02 23:25:09.177535 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.177444 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-credential-socket" pod "65ee46cd-bfd4-439b-a15c-394e7d10b915" (UID: "65ee46cd-bfd4-439b-a15c-394e7d10b915"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:09.177535 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.177494 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-istio-envoy\") pod \"65ee46cd-bfd4-439b-a15c-394e7d10b915\" (UID: \"65ee46cd-bfd4-439b-a15c-394e7d10b915\") " Nov 02 23:25:09.177535 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.177522 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-istio-data" pod "65ee46cd-bfd4-439b-a15c-394e7d10b915" (UID: "65ee46cd-bfd4-439b-a15c-394e7d10b915"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:09.177535 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.177532 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-workload-socket" pod "65ee46cd-bfd4-439b-a15c-394e7d10b915" (UID: "65ee46cd-bfd4-439b-a15c-394e7d10b915"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:09.177894 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.177751 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-workload-certs" pod "65ee46cd-bfd4-439b-a15c-394e7d10b915" (UID: "65ee46cd-bfd4-439b-a15c-394e7d10b915"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:09.177894 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.177800 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65ee46cd-bfd4-439b-a15c-394e7d10b915-istiod-ca-cert" pod "65ee46cd-bfd4-439b-a15c-394e7d10b915" (UID: "65ee46cd-bfd4-439b-a15c-394e7d10b915"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:25:09.178113 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.178093 5727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ee46cd-bfd4-439b-a15c-394e7d10b915-istiod-ca-cert\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:09.178379 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.178120 5727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-workload-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:09.178379 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.178160 5727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-credential-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:09.178379 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.178174 5727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-workload-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:09.178379 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.178188 5727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-istio-data\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:09.179978 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.179952 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-istio-envoy" pod "65ee46cd-bfd4-439b-a15c-394e7d10b915" (UID: "65ee46cd-bfd4-439b-a15c-394e7d10b915"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:09.180245 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.180216 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ee46cd-bfd4-439b-a15c-394e7d10b915-kube-api-access-nxdx8" pod "65ee46cd-bfd4-439b-a15c-394e7d10b915" (UID: "65ee46cd-bfd4-439b-a15c-394e7d10b915"). InnerVolumeSpecName "kube-api-access-nxdx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:25:09.180245 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.180232 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65ee46cd-bfd4-439b-a15c-394e7d10b915-istio-podinfo" pod "65ee46cd-bfd4-439b-a15c-394e7d10b915" (UID: "65ee46cd-bfd4-439b-a15c-394e7d10b915"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:25:09.180409 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.180392 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ee46cd-bfd4-439b-a15c-394e7d10b915-istio-token" pod "65ee46cd-bfd4-439b-a15c-394e7d10b915" (UID: "65ee46cd-bfd4-439b-a15c-394e7d10b915"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:25:09.280025 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.279936 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxdx8\" (UniqueName: \"kubernetes.io/projected/65ee46cd-bfd4-439b-a15c-394e7d10b915-kube-api-access-nxdx8\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:09.280025 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.279977 5727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65ee46cd-bfd4-439b-a15c-394e7d10b915-istio-token\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:09.280025 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.279991 5727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ee46cd-bfd4-439b-a15c-394e7d10b915-istio-podinfo\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:09.280025 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.280005 5727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ee46cd-bfd4-439b-a15c-394e7d10b915-istio-envoy\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:09.555818 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.555726 5727 generic.go:356] "Generic (PLEG): container finished" podID="65ee46cd-bfd4-439b-a15c-394e7d10b915" containerID="b9a106765c32a5935e454d98227962009e2693dd6e666a0557e50e196c12c2b6" exitCode=0 Nov 02 23:25:09.555818 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.555784 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" Nov 02 23:25:09.555818 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.555796 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" event={"ID":"65ee46cd-bfd4-439b-a15c-394e7d10b915","Type":"ContainerDied","Data":"b9a106765c32a5935e454d98227962009e2693dd6e666a0557e50e196c12c2b6"} Nov 02 23:25:09.556088 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.555869 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df" event={"ID":"65ee46cd-bfd4-439b-a15c-394e7d10b915","Type":"ContainerDied","Data":"30e5352269723ba7607109279a7290b5b9dcd09ee71387966b41cd9e8f408fc1"} Nov 02 23:25:09.556088 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.555899 5727 scope.go:122] "RemoveContainer" containerID="b9a106765c32a5935e454d98227962009e2693dd6e666a0557e50e196c12c2b6" Nov 02 23:25:09.567726 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.567707 5727 scope.go:122] "RemoveContainer" containerID="b9a106765c32a5935e454d98227962009e2693dd6e666a0557e50e196c12c2b6" Nov 02 23:25:09.568040 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:25:09.568023 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9a106765c32a5935e454d98227962009e2693dd6e666a0557e50e196c12c2b6\": container with ID starting with b9a106765c32a5935e454d98227962009e2693dd6e666a0557e50e196c12c2b6 not found: ID does not exist" containerID="b9a106765c32a5935e454d98227962009e2693dd6e666a0557e50e196c12c2b6" Nov 02 23:25:09.568105 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.568048 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9a106765c32a5935e454d98227962009e2693dd6e666a0557e50e196c12c2b6"} err="failed to get container status \"b9a106765c32a5935e454d98227962009e2693dd6e666a0557e50e196c12c2b6\": rpc error: code = NotFound desc = could not find container \"b9a106765c32a5935e454d98227962009e2693dd6e666a0557e50e196c12c2b6\": container with ID starting with b9a106765c32a5935e454d98227962009e2693dd6e666a0557e50e196c12c2b6 not found: ID does not exist" Nov 02 23:25:09.721240 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.721208 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df"] Nov 02 23:25:09.771498 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.771460 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7g7df"] Nov 02 23:25:09.885664 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:09.885627 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65ee46cd-bfd4-439b-a15c-394e7d10b915" path="/var/lib/kubelet/pods/65ee46cd-bfd4-439b-a15c-394e7d10b915/volumes" Nov 02 23:25:27.078323 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.078288 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f549-68b787d455-pcjmj"] Nov 02 23:25:27.079223 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.079202 5727 cpu_manager.go:414] "containerMap: removing container" podUID="3bd21a3d-ad0a-48db-80fa-37b40e05bb74" containerName="istio-proxy" Nov 02 23:25:27.079292 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.079225 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bd21a3d-ad0a-48db-80fa-37b40e05bb74" containerName="istio-proxy" Nov 02 23:25:27.079292 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.079239 5727 cpu_manager.go:414] "containerMap: removing container" podUID="96c11959-99bc-467a-a844-74fc983c126e" containerName="maas-api" Nov 02 23:25:27.079292 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.079249 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96c11959-99bc-467a-a844-74fc983c126e" containerName="maas-api" Nov 02 23:25:27.079388 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.079294 5727 cpu_manager.go:414] "containerMap: removing container" podUID="bfdaf98c-fc85-4457-8366-4a81aab5573a" containerName="istio-proxy" Nov 02 23:25:27.079388 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.079303 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfdaf98c-fc85-4457-8366-4a81aab5573a" containerName="istio-proxy" Nov 02 23:25:27.079388 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.079331 5727 cpu_manager.go:414] "containerMap: removing container" podUID="65ee46cd-bfd4-439b-a15c-394e7d10b915" containerName="istio-proxy" Nov 02 23:25:27.079388 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.079338 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65ee46cd-bfd4-439b-a15c-394e7d10b915" containerName="istio-proxy" Nov 02 23:25:27.079388 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.079349 5727 cpu_manager.go:414] "containerMap: removing container" podUID="40350d65-8ea1-4d78-882c-f7b75c61dccd" containerName="istio-proxy" Nov 02 23:25:27.079388 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.079357 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40350d65-8ea1-4d78-882c-f7b75c61dccd" containerName="istio-proxy" Nov 02 23:25:27.079568 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.079512 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfdaf98c-fc85-4457-8366-4a81aab5573a" containerName="istio-proxy" Nov 02 23:25:27.079568 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.079525 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="65ee46cd-bfd4-439b-a15c-394e7d10b915" containerName="istio-proxy" Nov 02 23:25:27.079568 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.079534 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="40350d65-8ea1-4d78-882c-f7b75c61dccd" containerName="istio-proxy" Nov 02 23:25:27.079568 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.079551 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bd21a3d-ad0a-48db-80fa-37b40e05bb74" containerName="istio-proxy" Nov 02 23:25:27.079568 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.079559 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="96c11959-99bc-467a-a844-74fc983c126e" containerName="maas-api" Nov 02 23:25:27.083662 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.083642 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f549-68b787d455-pcjmj" Nov 02 23:25:27.144407 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.138613 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f549-68b787d455-pcjmj"] Nov 02 23:25:27.185684 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.184471 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-a2f549-dockercfg-b9798\"" Nov 02 23:25:27.185684 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.184573 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-a2f549\"" Nov 02 23:25:27.202579 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.202190 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c56d8a2-f496-4fd2-acf6-8f27122bddae-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a2f549-68b787d455-pcjmj\" (UID: \"9c56d8a2-f496-4fd2-acf6-8f27122bddae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f549-68b787d455-pcjmj" Nov 02 23:25:27.202579 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.202385 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d2tb\" (UniqueName: \"kubernetes.io/projected/9c56d8a2-f496-4fd2-acf6-8f27122bddae-kube-api-access-9d2tb\") pod \"payload-pre-processing-e2e-praxis-b-a2f549-68b787d455-pcjmj\" (UID: \"9c56d8a2-f496-4fd2-acf6-8f27122bddae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f549-68b787d455-pcjmj" Nov 02 23:25:27.239657 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.239621 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a2f549-7c54bf87f5-6g6xt"] Nov 02 23:25:27.244494 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.244470 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f549-7c54bf87f5-6g6xt" Nov 02 23:25:27.294443 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.294398 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a2f549-7c54bf87f5-6g6xt"] Nov 02 23:25:27.303588 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.303553 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d2tb\" (UniqueName: \"kubernetes.io/projected/9c56d8a2-f496-4fd2-acf6-8f27122bddae-kube-api-access-9d2tb\") pod \"payload-pre-processing-e2e-praxis-b-a2f549-68b787d455-pcjmj\" (UID: \"9c56d8a2-f496-4fd2-acf6-8f27122bddae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f549-68b787d455-pcjmj" Nov 02 23:25:27.303742 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.303705 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c56d8a2-f496-4fd2-acf6-8f27122bddae-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a2f549-68b787d455-pcjmj\" (UID: \"9c56d8a2-f496-4fd2-acf6-8f27122bddae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f549-68b787d455-pcjmj" Nov 02 23:25:27.304668 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.304645 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c56d8a2-f496-4fd2-acf6-8f27122bddae-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a2f549-68b787d455-pcjmj\" (UID: \"9c56d8a2-f496-4fd2-acf6-8f27122bddae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f549-68b787d455-pcjmj" Nov 02 23:25:27.365605 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.365529 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d2tb\" (UniqueName: \"kubernetes.io/projected/9c56d8a2-f496-4fd2-acf6-8f27122bddae-kube-api-access-9d2tb\") pod \"payload-pre-processing-e2e-praxis-b-a2f549-68b787d455-pcjmj\" (UID: \"9c56d8a2-f496-4fd2-acf6-8f27122bddae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f549-68b787d455-pcjmj" Nov 02 23:25:27.402258 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.401808 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f549-68b787d455-pcjmj" Nov 02 23:25:27.404717 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.404687 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/629bc7d8-4ece-473f-98b0-22a2571cd5e5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a2f549-7c54bf87f5-6g6xt\" (UID: \"629bc7d8-4ece-473f-98b0-22a2571cd5e5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f549-7c54bf87f5-6g6xt" Nov 02 23:25:27.404887 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.404739 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbv92\" (UniqueName: \"kubernetes.io/projected/629bc7d8-4ece-473f-98b0-22a2571cd5e5-kube-api-access-vbv92\") pod \"payload-processing-e2e-praxis-b-a2f549-7c54bf87f5-6g6xt\" (UID: \"629bc7d8-4ece-473f-98b0-22a2571cd5e5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f549-7c54bf87f5-6g6xt" Nov 02 23:25:27.505834 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.505803 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/629bc7d8-4ece-473f-98b0-22a2571cd5e5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a2f549-7c54bf87f5-6g6xt\" (UID: \"629bc7d8-4ece-473f-98b0-22a2571cd5e5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f549-7c54bf87f5-6g6xt" Nov 02 23:25:27.506003 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.505846 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbv92\" (UniqueName: \"kubernetes.io/projected/629bc7d8-4ece-473f-98b0-22a2571cd5e5-kube-api-access-vbv92\") pod \"payload-processing-e2e-praxis-b-a2f549-7c54bf87f5-6g6xt\" (UID: \"629bc7d8-4ece-473f-98b0-22a2571cd5e5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f549-7c54bf87f5-6g6xt" Nov 02 23:25:27.506694 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.506654 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/629bc7d8-4ece-473f-98b0-22a2571cd5e5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a2f549-7c54bf87f5-6g6xt\" (UID: \"629bc7d8-4ece-473f-98b0-22a2571cd5e5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f549-7c54bf87f5-6g6xt" Nov 02 23:25:27.566208 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.566173 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbv92\" (UniqueName: \"kubernetes.io/projected/629bc7d8-4ece-473f-98b0-22a2571cd5e5-kube-api-access-vbv92\") pod \"payload-processing-e2e-praxis-b-a2f549-7c54bf87f5-6g6xt\" (UID: \"629bc7d8-4ece-473f-98b0-22a2571cd5e5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f549-7c54bf87f5-6g6xt" Nov 02 23:25:27.788050 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.788019 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f549-68b787d455-pcjmj"] Nov 02 23:25:27.789111 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:25:27.789083 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c56d8a2_f496_4fd2_acf6_8f27122bddae.slice/crio-7a1428c5ce7de48411b99266eeabb6a2829d6fe3e337c46b76b98190f9afca4e WatchSource:0}: Error finding container 7a1428c5ce7de48411b99266eeabb6a2829d6fe3e337c46b76b98190f9afca4e: Status 404 returned error can't find the container with id 7a1428c5ce7de48411b99266eeabb6a2829d6fe3e337c46b76b98190f9afca4e Nov 02 23:25:27.847543 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:25:27.847507 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:25:27.861920 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:27.861897 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f549-7c54bf87f5-6g6xt" Nov 02 23:25:28.241241 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:28.241217 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a2f549-7c54bf87f5-6g6xt"] Nov 02 23:25:28.242635 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:25:28.242609 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629bc7d8_4ece_473f_98b0_22a2571cd5e5.slice/crio-a7653cc558e458bd97aa3bf97289d9b9652211359f4b93dfe3513e3eb32fd536 WatchSource:0}: Error finding container a7653cc558e458bd97aa3bf97289d9b9652211359f4b93dfe3513e3eb32fd536: Status 404 returned error can't find the container with id a7653cc558e458bd97aa3bf97289d9b9652211359f4b93dfe3513e3eb32fd536 Nov 02 23:25:28.737672 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:28.737631 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f549-68b787d455-pcjmj" event={"ID":"9c56d8a2-f496-4fd2-acf6-8f27122bddae","Type":"ContainerStarted","Data":"7fc3e5e9aab41fde9fd2167e7fb59cbcd81a5297d7bcaa828c672f0f58f16bcf"} Nov 02 23:25:28.737863 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:28.737838 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f549-68b787d455-pcjmj" Nov 02 23:25:28.737926 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:28.737881 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f549-68b787d455-pcjmj" event={"ID":"9c56d8a2-f496-4fd2-acf6-8f27122bddae","Type":"ContainerStarted","Data":"7a1428c5ce7de48411b99266eeabb6a2829d6fe3e337c46b76b98190f9afca4e"} Nov 02 23:25:28.739512 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:28.739488 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f549-7c54bf87f5-6g6xt" event={"ID":"629bc7d8-4ece-473f-98b0-22a2571cd5e5","Type":"ContainerStarted","Data":"be4afe45ccb9a0c55d449b79968e0b39c7d1aa60852949800adcb0a095bb4862"} Nov 02 23:25:28.739632 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:28.739523 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f549-7c54bf87f5-6g6xt" event={"ID":"629bc7d8-4ece-473f-98b0-22a2571cd5e5","Type":"ContainerStarted","Data":"a7653cc558e458bd97aa3bf97289d9b9652211359f4b93dfe3513e3eb32fd536"} Nov 02 23:25:28.739632 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:28.739599 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f549-7c54bf87f5-6g6xt" Nov 02 23:25:28.855404 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:28.855120 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f549-68b787d455-pcjmj" podStartSLOduration=1.855100878 podStartE2EDuration="1.855100878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:25:28.852052742 +0000 UTC m=+2215.090550430" watchObservedRunningTime="2026-11-02 23:25:28.855100878 +0000 UTC m=+2215.093598546" Nov 02 23:25:28.973395 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:28.973316 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f549-7c54bf87f5-6g6xt" podStartSLOduration=1.973290886 podStartE2EDuration="1.973290886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:25:28.967383415 +0000 UTC m=+2215.205881107" watchObservedRunningTime="2026-11-02 23:25:28.973290886 +0000 UTC m=+2215.211788551" Nov 02 23:25:39.750917 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:39.750883 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f549-68b787d455-pcjmj" Nov 02 23:25:39.751449 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:39.750960 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f549-7c54bf87f5-6g6xt" Nov 02 23:25:43.996379 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:43.996334 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-85efc6-7dd5465b9f-t426x"] Nov 02 23:25:44.003349 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:44.003327 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85efc6-7dd5465b9f-t426x" Nov 02 23:25:44.055957 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:44.055926 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-85efc6-7dd5465b9f-t426x"] Nov 02 23:25:44.101360 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:44.101328 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-85efc6\"" Nov 02 23:25:44.101757 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:44.101736 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-85efc6-dockercfg-4pq9b\"" Nov 02 23:25:44.126369 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:44.126334 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdwcx\" (UniqueName: \"kubernetes.io/projected/c559df9f-192e-47d0-8b88-e13ff6259fd6-kube-api-access-jdwcx\") pod \"payload-pre-processing-e2e-praxis-b-85efc6-7dd5465b9f-t426x\" (UID: \"c559df9f-192e-47d0-8b88-e13ff6259fd6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85efc6-7dd5465b9f-t426x" Nov 02 23:25:44.126591 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:44.126409 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c559df9f-192e-47d0-8b88-e13ff6259fd6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-85efc6-7dd5465b9f-t426x\" (UID: \"c559df9f-192e-47d0-8b88-e13ff6259fd6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85efc6-7dd5465b9f-t426x" Nov 02 23:25:44.155716 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:44.155678 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-85efc6-74998f459-r4dgk"] Nov 02 23:25:44.161189 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:44.161123 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-85efc6-74998f459-r4dgk" Nov 02 23:25:44.205025 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:44.204993 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-85efc6-74998f459-r4dgk"] Nov 02 23:25:44.227734 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:44.227704 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdwcx\" (UniqueName: \"kubernetes.io/projected/c559df9f-192e-47d0-8b88-e13ff6259fd6-kube-api-access-jdwcx\") pod \"payload-pre-processing-e2e-praxis-b-85efc6-7dd5465b9f-t426x\" (UID: \"c559df9f-192e-47d0-8b88-e13ff6259fd6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85efc6-7dd5465b9f-t426x" Nov 02 23:25:44.227925 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:44.227756 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c559df9f-192e-47d0-8b88-e13ff6259fd6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-85efc6-7dd5465b9f-t426x\" (UID: \"c559df9f-192e-47d0-8b88-e13ff6259fd6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85efc6-7dd5465b9f-t426x" Nov 02 23:25:44.227925 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:44.227886 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c53de2f2-1902-4cd3-94d6-2f06ab70aa69-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-85efc6-74998f459-r4dgk\" (UID: \"c53de2f2-1902-4cd3-94d6-2f06ab70aa69\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-85efc6-74998f459-r4dgk" Nov 02 23:25:44.228043 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:44.227950 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnktz\" (UniqueName: \"kubernetes.io/projected/c53de2f2-1902-4cd3-94d6-2f06ab70aa69-kube-api-access-pnktz\") pod \"payload-processing-e2e-praxis-b-85efc6-74998f459-r4dgk\" (UID: \"c53de2f2-1902-4cd3-94d6-2f06ab70aa69\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-85efc6-74998f459-r4dgk" Nov 02 23:25:44.228657 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:44.228632 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c559df9f-192e-47d0-8b88-e13ff6259fd6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-85efc6-7dd5465b9f-t426x\" (UID: \"c559df9f-192e-47d0-8b88-e13ff6259fd6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85efc6-7dd5465b9f-t426x" Nov 02 23:25:44.288355 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:44.288277 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdwcx\" (UniqueName: \"kubernetes.io/projected/c559df9f-192e-47d0-8b88-e13ff6259fd6-kube-api-access-jdwcx\") pod \"payload-pre-processing-e2e-praxis-b-85efc6-7dd5465b9f-t426x\" (UID: \"c559df9f-192e-47d0-8b88-e13ff6259fd6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85efc6-7dd5465b9f-t426x" Nov 02 23:25:44.319298 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:44.319268 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85efc6-7dd5465b9f-t426x" Nov 02 23:25:44.329121 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:44.329092 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c53de2f2-1902-4cd3-94d6-2f06ab70aa69-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-85efc6-74998f459-r4dgk\" (UID: \"c53de2f2-1902-4cd3-94d6-2f06ab70aa69\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-85efc6-74998f459-r4dgk" Nov 02 23:25:44.329285 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:44.329174 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnktz\" (UniqueName: \"kubernetes.io/projected/c53de2f2-1902-4cd3-94d6-2f06ab70aa69-kube-api-access-pnktz\") pod \"payload-processing-e2e-praxis-b-85efc6-74998f459-r4dgk\" (UID: \"c53de2f2-1902-4cd3-94d6-2f06ab70aa69\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-85efc6-74998f459-r4dgk" Nov 02 23:25:44.329977 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:44.329956 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c53de2f2-1902-4cd3-94d6-2f06ab70aa69-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-85efc6-74998f459-r4dgk\" (UID: \"c53de2f2-1902-4cd3-94d6-2f06ab70aa69\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-85efc6-74998f459-r4dgk" Nov 02 23:25:44.411185 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:44.409155 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnktz\" (UniqueName: \"kubernetes.io/projected/c53de2f2-1902-4cd3-94d6-2f06ab70aa69-kube-api-access-pnktz\") pod \"payload-processing-e2e-praxis-b-85efc6-74998f459-r4dgk\" (UID: \"c53de2f2-1902-4cd3-94d6-2f06ab70aa69\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-85efc6-74998f459-r4dgk" Nov 02 23:25:44.477530 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:44.477494 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-85efc6-74998f459-r4dgk" Nov 02 23:25:44.704164 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:44.704118 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-85efc6-7dd5465b9f-t426x"] Nov 02 23:25:44.706055 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:25:44.706025 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc559df9f_192e_47d0_8b88_e13ff6259fd6.slice/crio-731ced98204f520efb83d8fe4030c007065d6545c62d8e13b8f9c0db2d623f00 WatchSource:0}: Error finding container 731ced98204f520efb83d8fe4030c007065d6545c62d8e13b8f9c0db2d623f00: Status 404 returned error can't find the container with id 731ced98204f520efb83d8fe4030c007065d6545c62d8e13b8f9c0db2d623f00 Nov 02 23:25:44.864239 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:44.864206 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-85efc6-74998f459-r4dgk"] Nov 02 23:25:44.864389 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:25:44.864221 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53de2f2_1902_4cd3_94d6_2f06ab70aa69.slice/crio-5c650cbeeff1f509027d483393ba7abdc40129a0924308124f29eebf10dfe107 WatchSource:0}: Error finding container 5c650cbeeff1f509027d483393ba7abdc40129a0924308124f29eebf10dfe107: Status 404 returned error can't find the container with id 5c650cbeeff1f509027d483393ba7abdc40129a0924308124f29eebf10dfe107 Nov 02 23:25:44.884743 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:44.884710 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85efc6-7dd5465b9f-t426x" event={"ID":"c559df9f-192e-47d0-8b88-e13ff6259fd6","Type":"ContainerStarted","Data":"bc0419cb75b2ce8265ca3131615650ac4ee8cb45bef6fc9e8be26a5afee5f42e"} Nov 02 23:25:44.884899 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:44.884770 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85efc6-7dd5465b9f-t426x" event={"ID":"c559df9f-192e-47d0-8b88-e13ff6259fd6","Type":"ContainerStarted","Data":"731ced98204f520efb83d8fe4030c007065d6545c62d8e13b8f9c0db2d623f00"} Nov 02 23:25:44.884899 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:44.884799 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85efc6-7dd5465b9f-t426x" Nov 02 23:25:44.886537 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:44.886511 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-85efc6-74998f459-r4dgk" event={"ID":"c53de2f2-1902-4cd3-94d6-2f06ab70aa69","Type":"ContainerStarted","Data":"5c650cbeeff1f509027d483393ba7abdc40129a0924308124f29eebf10dfe107"} Nov 02 23:25:45.002537 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:45.002477 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85efc6-7dd5465b9f-t426x" podStartSLOduration=2.002459881 podStartE2EDuration="2.002459881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:25:44.999104478 +0000 UTC m=+2231.237602142" watchObservedRunningTime="2026-11-02 23:25:45.002459881 +0000 UTC m=+2231.240957544" Nov 02 23:25:45.895932 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:45.895881 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-85efc6-74998f459-r4dgk" event={"ID":"c53de2f2-1902-4cd3-94d6-2f06ab70aa69","Type":"ContainerStarted","Data":"63af311d74a2d0ae05ed342a326aa70173a27cda7118e0d0291514c0c8a237cc"} Nov 02 23:25:45.896270 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:45.896245 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-85efc6-74998f459-r4dgk" Nov 02 23:25:46.013008 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:46.012947 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-85efc6-74998f459-r4dgk" podStartSLOduration=2.012933102 podStartE2EDuration="2.012933102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:25:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:25:46.010485693 +0000 UTC m=+2232.248983379" watchObservedRunningTime="2026-11-02 23:25:46.012933102 +0000 UTC m=+2232.251430763" Nov 02 23:25:55.899683 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:55.899649 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85efc6-7dd5465b9f-t426x" Nov 02 23:25:56.907246 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:25:56.907217 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-85efc6-74998f459-r4dgk" Nov 02 23:25:57.847381 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:25:57.847342 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:26:00.598641 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:00.598601 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7549c7cff-j4lm4"] Nov 02 23:26:00.703957 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:00.703919 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-5674cb547f-cgtjh"] Nov 02 23:26:05.300913 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:05.300878 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e875bc-5f4d6557b7-ct7tk"] Nov 02 23:26:05.598968 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:05.598905 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7549c7cff-j4lm4" podUID="ea5fa62a-c0d2-412c-ac94-5b5f4482b110" containerName="payload-processing" containerID="cri-o://01231fb3643cf7a094cc482b55f1966e8b3dc032644203baa597809484081e7f" gracePeriod=25 Nov 02 23:26:05.705169 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:05.705089 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-5674cb547f-cgtjh" podUID="63e2eb91-30e8-49e7-8b0a-9a66435e46e8" containerName="payload-pre-processing" containerID="cri-o://bf5ddc9a741fa2b1010aa119aa19c828bd94bdb55cc2006a5fb6cb5d2ac2657b" gracePeriod=25 Nov 02 23:26:06.001727 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:06.001694 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7549c7cff-j4lm4" Nov 02 23:26:06.052890 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:06.052862 5727 generic.go:356] "Generic (PLEG): container finished" podID="63e2eb91-30e8-49e7-8b0a-9a66435e46e8" containerID="bf5ddc9a741fa2b1010aa119aa19c828bd94bdb55cc2006a5fb6cb5d2ac2657b" exitCode=0 Nov 02 23:26:06.053018 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:06.052935 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-5674cb547f-cgtjh" event={"ID":"63e2eb91-30e8-49e7-8b0a-9a66435e46e8","Type":"ContainerDied","Data":"bf5ddc9a741fa2b1010aa119aa19c828bd94bdb55cc2006a5fb6cb5d2ac2657b"} Nov 02 23:26:06.053018 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:06.052996 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-5674cb547f-cgtjh" event={"ID":"63e2eb91-30e8-49e7-8b0a-9a66435e46e8","Type":"ContainerDied","Data":"37fd273988d326f8611c4677c95c92b67b499cd097c2afb52bd2cd5dc4242311"} Nov 02 23:26:06.053117 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:06.053020 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37fd273988d326f8611c4677c95c92b67b499cd097c2afb52bd2cd5dc4242311" Nov 02 23:26:06.054723 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:06.054699 5727 generic.go:356] "Generic (PLEG): container finished" podID="ea5fa62a-c0d2-412c-ac94-5b5f4482b110" containerID="01231fb3643cf7a094cc482b55f1966e8b3dc032644203baa597809484081e7f" exitCode=0 Nov 02 23:26:06.054860 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:06.054755 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7549c7cff-j4lm4" Nov 02 23:26:06.054860 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:06.054778 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7549c7cff-j4lm4" event={"ID":"ea5fa62a-c0d2-412c-ac94-5b5f4482b110","Type":"ContainerDied","Data":"01231fb3643cf7a094cc482b55f1966e8b3dc032644203baa597809484081e7f"} Nov 02 23:26:06.054860 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:06.054833 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7549c7cff-j4lm4" event={"ID":"ea5fa62a-c0d2-412c-ac94-5b5f4482b110","Type":"ContainerDied","Data":"1ddb372f0a09631e3d063c8167be4c4550080491dfee5fc6b8abed85af7d82df"} Nov 02 23:26:06.054982 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:06.054862 5727 scope.go:122] "RemoveContainer" containerID="01231fb3643cf7a094cc482b55f1966e8b3dc032644203baa597809484081e7f" Nov 02 23:26:06.059769 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:06.059749 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-5674cb547f-cgtjh" Nov 02 23:26:06.066108 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:06.066087 5727 scope.go:122] "RemoveContainer" containerID="01231fb3643cf7a094cc482b55f1966e8b3dc032644203baa597809484081e7f" Nov 02 23:26:06.066440 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:26:06.066416 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01231fb3643cf7a094cc482b55f1966e8b3dc032644203baa597809484081e7f\": container with ID starting with 01231fb3643cf7a094cc482b55f1966e8b3dc032644203baa597809484081e7f not found: ID does not exist" containerID="01231fb3643cf7a094cc482b55f1966e8b3dc032644203baa597809484081e7f" Nov 02 23:26:06.066494 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:06.066452 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01231fb3643cf7a094cc482b55f1966e8b3dc032644203baa597809484081e7f"} err="failed to get container status \"01231fb3643cf7a094cc482b55f1966e8b3dc032644203baa597809484081e7f\": rpc error: code = NotFound desc = could not find container \"01231fb3643cf7a094cc482b55f1966e8b3dc032644203baa597809484081e7f\": container with ID starting with 01231fb3643cf7a094cc482b55f1966e8b3dc032644203baa597809484081e7f not found: ID does not exist" Nov 02 23:26:06.114488 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:06.114454 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea5fa62a-c0d2-412c-ac94-5b5f4482b110-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5fa62a-c0d2-412c-ac94-5b5f4482b110-plugins-config-volume\") pod \"ea5fa62a-c0d2-412c-ac94-5b5f4482b110\" (UID: \"ea5fa62a-c0d2-412c-ac94-5b5f4482b110\") " Nov 02 23:26:06.114657 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:06.114621 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63e2eb91-30e8-49e7-8b0a-9a66435e46e8-kube-api-access-fzq6k\" (UniqueName: \"kubernetes.io/projected/63e2eb91-30e8-49e7-8b0a-9a66435e46e8-kube-api-access-fzq6k\") pod \"63e2eb91-30e8-49e7-8b0a-9a66435e46e8\" (UID: \"63e2eb91-30e8-49e7-8b0a-9a66435e46e8\") " Nov 02 23:26:06.114720 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:06.114678 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea5fa62a-c0d2-412c-ac94-5b5f4482b110-kube-api-access-rbvxv\" (UniqueName: \"kubernetes.io/projected/ea5fa62a-c0d2-412c-ac94-5b5f4482b110-kube-api-access-rbvxv\") pod \"ea5fa62a-c0d2-412c-ac94-5b5f4482b110\" (UID: \"ea5fa62a-c0d2-412c-ac94-5b5f4482b110\") " Nov 02 23:26:06.114801 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:06.114781 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63e2eb91-30e8-49e7-8b0a-9a66435e46e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63e2eb91-30e8-49e7-8b0a-9a66435e46e8-plugins-config-volume\") pod \"63e2eb91-30e8-49e7-8b0a-9a66435e46e8\" (UID: \"63e2eb91-30e8-49e7-8b0a-9a66435e46e8\") " Nov 02 23:26:06.114946 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:06.114923 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea5fa62a-c0d2-412c-ac94-5b5f4482b110-plugins-config-volume" pod "ea5fa62a-c0d2-412c-ac94-5b5f4482b110" (UID: "ea5fa62a-c0d2-412c-ac94-5b5f4482b110"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:26:06.115315 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:06.115287 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5fa62a-c0d2-412c-ac94-5b5f4482b110-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:26:06.115315 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:06.115288 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63e2eb91-30e8-49e7-8b0a-9a66435e46e8-plugins-config-volume" pod "63e2eb91-30e8-49e7-8b0a-9a66435e46e8" (UID: "63e2eb91-30e8-49e7-8b0a-9a66435e46e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:26:06.117533 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:06.117484 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5fa62a-c0d2-412c-ac94-5b5f4482b110-kube-api-access-rbvxv" pod "ea5fa62a-c0d2-412c-ac94-5b5f4482b110" (UID: "ea5fa62a-c0d2-412c-ac94-5b5f4482b110"). InnerVolumeSpecName "kube-api-access-rbvxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:26:06.117635 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:06.117603 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63e2eb91-30e8-49e7-8b0a-9a66435e46e8-kube-api-access-fzq6k" pod "63e2eb91-30e8-49e7-8b0a-9a66435e46e8" (UID: "63e2eb91-30e8-49e7-8b0a-9a66435e46e8"). InnerVolumeSpecName "kube-api-access-fzq6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:26:06.216711 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:06.216615 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzq6k\" (UniqueName: \"kubernetes.io/projected/63e2eb91-30e8-49e7-8b0a-9a66435e46e8-kube-api-access-fzq6k\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:26:06.216711 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:06.216649 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbvxv\" (UniqueName: \"kubernetes.io/projected/ea5fa62a-c0d2-412c-ac94-5b5f4482b110-kube-api-access-rbvxv\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:26:06.216711 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:06.216665 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63e2eb91-30e8-49e7-8b0a-9a66435e46e8-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:26:06.479560 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:06.479443 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7549c7cff-j4lm4"] Nov 02 23:26:06.482960 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:06.482938 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7549c7cff-j4lm4"] Nov 02 23:26:07.066521 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:07.066493 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-5674cb547f-cgtjh" Nov 02 23:26:07.231203 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:07.231169 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-5674cb547f-cgtjh"] Nov 02 23:26:07.234353 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:07.234334 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-5674cb547f-cgtjh"] Nov 02 23:26:07.883506 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:07.883473 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63e2eb91-30e8-49e7-8b0a-9a66435e46e8" path="/var/lib/kubelet/pods/63e2eb91-30e8-49e7-8b0a-9a66435e46e8/volumes" Nov 02 23:26:07.883903 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:07.883886 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea5fa62a-c0d2-412c-ac94-5b5f4482b110" path="/var/lib/kubelet/pods/ea5fa62a-c0d2-412c-ac94-5b5f4482b110/volumes" Nov 02 23:26:10.302182 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:10.302112 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e875bc-5f4d6557b7-ct7tk" podUID="fde5fb1f-bd36-45b0-83fa-b3be9e144340" containerName="payload-pre-processing" containerID="cri-o://ee2244f3b868019d45e238968ba5efea1beaf13a617c7efa8d75926c558f3dcd" gracePeriod=25 Nov 02 23:26:10.593230 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:10.592822 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e875bc-5f4d6557b7-ct7tk" Nov 02 23:26:10.681892 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:10.681859 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fde5fb1f-bd36-45b0-83fa-b3be9e144340-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fde5fb1f-bd36-45b0-83fa-b3be9e144340-plugins-config-volume\") pod \"fde5fb1f-bd36-45b0-83fa-b3be9e144340\" (UID: \"fde5fb1f-bd36-45b0-83fa-b3be9e144340\") " Nov 02 23:26:10.682220 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:10.681940 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fde5fb1f-bd36-45b0-83fa-b3be9e144340-kube-api-access-v5zk9\" (UniqueName: \"kubernetes.io/projected/fde5fb1f-bd36-45b0-83fa-b3be9e144340-kube-api-access-v5zk9\") pod \"fde5fb1f-bd36-45b0-83fa-b3be9e144340\" (UID: \"fde5fb1f-bd36-45b0-83fa-b3be9e144340\") " Nov 02 23:26:10.682333 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:10.682304 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fde5fb1f-bd36-45b0-83fa-b3be9e144340-plugins-config-volume" pod "fde5fb1f-bd36-45b0-83fa-b3be9e144340" (UID: "fde5fb1f-bd36-45b0-83fa-b3be9e144340"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:26:10.682497 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:10.682480 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fde5fb1f-bd36-45b0-83fa-b3be9e144340-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:26:10.684449 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:10.684428 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fde5fb1f-bd36-45b0-83fa-b3be9e144340-kube-api-access-v5zk9" pod "fde5fb1f-bd36-45b0-83fa-b3be9e144340" (UID: "fde5fb1f-bd36-45b0-83fa-b3be9e144340"). InnerVolumeSpecName "kube-api-access-v5zk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:26:10.783719 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:10.783680 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5zk9\" (UniqueName: \"kubernetes.io/projected/fde5fb1f-bd36-45b0-83fa-b3be9e144340-kube-api-access-v5zk9\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:26:11.104512 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:11.104473 5727 generic.go:356] "Generic (PLEG): container finished" podID="fde5fb1f-bd36-45b0-83fa-b3be9e144340" containerID="ee2244f3b868019d45e238968ba5efea1beaf13a617c7efa8d75926c558f3dcd" exitCode=0 Nov 02 23:26:11.104714 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:11.104518 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e875bc-5f4d6557b7-ct7tk" Nov 02 23:26:11.104714 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:11.104558 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e875bc-5f4d6557b7-ct7tk" event={"ID":"fde5fb1f-bd36-45b0-83fa-b3be9e144340","Type":"ContainerDied","Data":"ee2244f3b868019d45e238968ba5efea1beaf13a617c7efa8d75926c558f3dcd"} Nov 02 23:26:11.104714 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:11.104616 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e875bc-5f4d6557b7-ct7tk" event={"ID":"fde5fb1f-bd36-45b0-83fa-b3be9e144340","Type":"ContainerDied","Data":"f009efd9b8f85feecf812fc12db950b3dea39c40024b9823f4d3bf0d12c1c106"} Nov 02 23:26:11.104714 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:11.104643 5727 scope.go:122] "RemoveContainer" containerID="ee2244f3b868019d45e238968ba5efea1beaf13a617c7efa8d75926c558f3dcd" Nov 02 23:26:11.115743 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:11.115723 5727 scope.go:122] "RemoveContainer" containerID="ee2244f3b868019d45e238968ba5efea1beaf13a617c7efa8d75926c558f3dcd" Nov 02 23:26:11.116024 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:26:11.116002 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee2244f3b868019d45e238968ba5efea1beaf13a617c7efa8d75926c558f3dcd\": container with ID starting with ee2244f3b868019d45e238968ba5efea1beaf13a617c7efa8d75926c558f3dcd not found: ID does not exist" containerID="ee2244f3b868019d45e238968ba5efea1beaf13a617c7efa8d75926c558f3dcd" Nov 02 23:26:11.116085 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:11.116035 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee2244f3b868019d45e238968ba5efea1beaf13a617c7efa8d75926c558f3dcd"} err="failed to get container status \"ee2244f3b868019d45e238968ba5efea1beaf13a617c7efa8d75926c558f3dcd\": rpc error: code = NotFound desc = could not find container \"ee2244f3b868019d45e238968ba5efea1beaf13a617c7efa8d75926c558f3dcd\": container with ID starting with ee2244f3b868019d45e238968ba5efea1beaf13a617c7efa8d75926c558f3dcd not found: ID does not exist" Nov 02 23:26:11.270430 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:11.270361 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e875bc-5f4d6557b7-ct7tk"] Nov 02 23:26:11.275603 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:11.275576 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e875bc-5f4d6557b7-ct7tk"] Nov 02 23:26:11.885917 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:11.885878 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fde5fb1f-bd36-45b0-83fa-b3be9e144340" path="/var/lib/kubelet/pods/fde5fb1f-bd36-45b0-83fa-b3be9e144340/volumes" Nov 02 23:26:27.847075 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:26:27.847031 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:26:52.543205 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:52.543155 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4"] Nov 02 23:26:52.545474 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:52.543826 5727 cpu_manager.go:414] "containerMap: removing container" podUID="ea5fa62a-c0d2-412c-ac94-5b5f4482b110" containerName="payload-processing" Nov 02 23:26:52.545474 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:52.543839 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea5fa62a-c0d2-412c-ac94-5b5f4482b110" containerName="payload-processing" Nov 02 23:26:52.545474 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:52.543854 5727 cpu_manager.go:414] "containerMap: removing container" podUID="63e2eb91-30e8-49e7-8b0a-9a66435e46e8" containerName="payload-pre-processing" Nov 02 23:26:52.545474 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:52.543859 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63e2eb91-30e8-49e7-8b0a-9a66435e46e8" containerName="payload-pre-processing" Nov 02 23:26:52.545474 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:52.543892 5727 cpu_manager.go:414] "containerMap: removing container" podUID="fde5fb1f-bd36-45b0-83fa-b3be9e144340" containerName="payload-pre-processing" Nov 02 23:26:52.545474 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:52.543897 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fde5fb1f-bd36-45b0-83fa-b3be9e144340" containerName="payload-pre-processing" Nov 02 23:26:52.545474 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:52.543995 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea5fa62a-c0d2-412c-ac94-5b5f4482b110" containerName="payload-processing" Nov 02 23:26:52.545474 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:52.544002 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="63e2eb91-30e8-49e7-8b0a-9a66435e46e8" containerName="payload-pre-processing" Nov 02 23:26:52.545474 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:52.544015 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="fde5fb1f-bd36-45b0-83fa-b3be9e144340" containerName="payload-pre-processing" Nov 02 23:26:52.547541 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:52.547523 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4" Nov 02 23:26:52.601260 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:52.601223 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b57e0c31-0e6f-456a-9855-cb5d9c083c88-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4\" (UID: \"b57e0c31-0e6f-456a-9855-cb5d9c083c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4" Nov 02 23:26:52.601607 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:52.601575 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b57e0c31-0e6f-456a-9855-cb5d9c083c88-maas-api-tls\") pod \"maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4\" (UID: \"b57e0c31-0e6f-456a-9855-cb5d9c083c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4" Nov 02 23:26:52.601765 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:52.601667 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b57e0c31-0e6f-456a-9855-cb5d9c083c88-metrics-certs\") pod \"maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4\" (UID: \"b57e0c31-0e6f-456a-9855-cb5d9c083c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4" Nov 02 23:26:52.601765 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:52.601754 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxd27\" (UniqueName: \"kubernetes.io/projected/b57e0c31-0e6f-456a-9855-cb5d9c083c88-kube-api-access-mxd27\") pod \"maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4\" (UID: \"b57e0c31-0e6f-456a-9855-cb5d9c083c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4" Nov 02 23:26:52.651025 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:52.650988 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-85efc6-serving-cert\"" Nov 02 23:26:52.656490 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:52.656458 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4"] Nov 02 23:26:52.703078 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:52.703035 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b57e0c31-0e6f-456a-9855-cb5d9c083c88-metrics-certs\") pod \"maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4\" (UID: \"b57e0c31-0e6f-456a-9855-cb5d9c083c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4" Nov 02 23:26:52.703352 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:52.703200 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxd27\" (UniqueName: \"kubernetes.io/projected/b57e0c31-0e6f-456a-9855-cb5d9c083c88-kube-api-access-mxd27\") pod \"maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4\" (UID: \"b57e0c31-0e6f-456a-9855-cb5d9c083c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4" Nov 02 23:26:52.703352 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:52.703244 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b57e0c31-0e6f-456a-9855-cb5d9c083c88-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4\" (UID: \"b57e0c31-0e6f-456a-9855-cb5d9c083c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4" Nov 02 23:26:52.703527 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:52.703372 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b57e0c31-0e6f-456a-9855-cb5d9c083c88-maas-api-tls\") pod \"maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4\" (UID: \"b57e0c31-0e6f-456a-9855-cb5d9c083c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4" Nov 02 23:26:52.703585 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:26:52.703557 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-85efc6-serving-cert: secret "maas-api-e2e-praxis-b-85efc6-serving-cert" not found Nov 02 23:26:52.703638 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:26:52.703618 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b57e0c31-0e6f-456a-9855-cb5d9c083c88-maas-api-tls podName:b57e0c31-0e6f-456a-9855-cb5d9c083c88 nodeName:}" failed. No retries permitted until 2026-11-02 23:26:53.203603269 +0000 UTC m=+2299.442100922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b57e0c31-0e6f-456a-9855-cb5d9c083c88-maas-api-tls") pod "maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4" (UID: "b57e0c31-0e6f-456a-9855-cb5d9c083c88") : secret "maas-api-e2e-praxis-b-85efc6-serving-cert" not found Nov 02 23:26:52.704259 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:52.704219 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b57e0c31-0e6f-456a-9855-cb5d9c083c88-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4\" (UID: \"b57e0c31-0e6f-456a-9855-cb5d9c083c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4" Nov 02 23:26:52.706186 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:52.706122 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b57e0c31-0e6f-456a-9855-cb5d9c083c88-metrics-certs\") pod \"maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4\" (UID: \"b57e0c31-0e6f-456a-9855-cb5d9c083c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4" Nov 02 23:26:52.764164 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:52.764121 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxd27\" (UniqueName: \"kubernetes.io/projected/b57e0c31-0e6f-456a-9855-cb5d9c083c88-kube-api-access-mxd27\") pod \"maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4\" (UID: \"b57e0c31-0e6f-456a-9855-cb5d9c083c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4" Nov 02 23:26:53.210890 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:53.210852 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b57e0c31-0e6f-456a-9855-cb5d9c083c88-maas-api-tls\") pod \"maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4\" (UID: \"b57e0c31-0e6f-456a-9855-cb5d9c083c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4" Nov 02 23:26:53.213594 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:53.213570 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b57e0c31-0e6f-456a-9855-cb5d9c083c88-maas-api-tls\") pod \"maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4\" (UID: \"b57e0c31-0e6f-456a-9855-cb5d9c083c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4" Nov 02 23:26:53.465425 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:53.465337 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4" Nov 02 23:26:53.848220 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:53.848191 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4"] Nov 02 23:26:53.849583 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:26:53.849549 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb57e0c31_0e6f_456a_9855_cb5d9c083c88.slice/crio-3955e2a1a8ad5654b23eebb92c3c0f11ef84564adba8fc6c1c126dab53f8a152 WatchSource:0}: Error finding container 3955e2a1a8ad5654b23eebb92c3c0f11ef84564adba8fc6c1c126dab53f8a152: Status 404 returned error can't find the container with id 3955e2a1a8ad5654b23eebb92c3c0f11ef84564adba8fc6c1c126dab53f8a152 Nov 02 23:26:54.449041 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:54.449005 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4" event={"ID":"b57e0c31-0e6f-456a-9855-cb5d9c083c88","Type":"ContainerStarted","Data":"3955e2a1a8ad5654b23eebb92c3c0f11ef84564adba8fc6c1c126dab53f8a152"} Nov 02 23:26:55.460707 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:55.460671 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4" event={"ID":"b57e0c31-0e6f-456a-9855-cb5d9c083c88","Type":"ContainerStarted","Data":"3629ebb2c14fa7dba37dd92396eb48aa5888cbc4b34325b8360392d6b4ada486"} Nov 02 23:26:55.461211 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:55.460895 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4" Nov 02 23:26:55.578687 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:26:55.578625 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4" podStartSLOduration=2.600537101 podStartE2EDuration="3.578606939s" totalImagesPullingTime="978.069838ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:26:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:26:55.578104322 +0000 UTC m=+2301.816601984" watchObservedRunningTime="2026-11-02 23:26:55.578606939 +0000 UTC m=+2301.817104602" Nov 02 23:26:57.847911 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:26:57.847872 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:27:01.478089 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:01.478061 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4" Nov 02 23:27:09.180148 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:09.180091 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf"] Nov 02 23:27:09.180666 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:09.180351 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf" podUID="9da6b802-026a-4118-9833-4802930688d8" containerName="maas-api" containerID="cri-o://fb20c56d24a7d0f90440abcdf9ce95c6e9f3e0f7b3dc4ccab779fcc153276338" gracePeriod=30 Nov 02 23:27:09.480809 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:09.480783 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf" Nov 02 23:27:09.597732 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:09.596543 5727 generic.go:356] "Generic (PLEG): container finished" podID="9da6b802-026a-4118-9833-4802930688d8" containerID="fb20c56d24a7d0f90440abcdf9ce95c6e9f3e0f7b3dc4ccab779fcc153276338" exitCode=0 Nov 02 23:27:09.597732 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:09.596644 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf" event={"ID":"9da6b802-026a-4118-9833-4802930688d8","Type":"ContainerDied","Data":"fb20c56d24a7d0f90440abcdf9ce95c6e9f3e0f7b3dc4ccab779fcc153276338"} Nov 02 23:27:09.597732 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:09.596699 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf" event={"ID":"9da6b802-026a-4118-9833-4802930688d8","Type":"ContainerDied","Data":"5475ee1677b49ad6d5e189097952a6883c4b7237643726e32bc013f4d1a90ad4"} Nov 02 23:27:09.597732 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:09.596728 5727 scope.go:122] "RemoveContainer" containerID="fb20c56d24a7d0f90440abcdf9ce95c6e9f3e0f7b3dc4ccab779fcc153276338" Nov 02 23:27:09.597732 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:09.596752 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf" Nov 02 23:27:09.613176 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:09.613138 5727 scope.go:122] "RemoveContainer" containerID="fb20c56d24a7d0f90440abcdf9ce95c6e9f3e0f7b3dc4ccab779fcc153276338" Nov 02 23:27:09.613532 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:27:09.613508 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb20c56d24a7d0f90440abcdf9ce95c6e9f3e0f7b3dc4ccab779fcc153276338\": container with ID starting with fb20c56d24a7d0f90440abcdf9ce95c6e9f3e0f7b3dc4ccab779fcc153276338 not found: ID does not exist" containerID="fb20c56d24a7d0f90440abcdf9ce95c6e9f3e0f7b3dc4ccab779fcc153276338" Nov 02 23:27:09.613610 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:09.613544 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb20c56d24a7d0f90440abcdf9ce95c6e9f3e0f7b3dc4ccab779fcc153276338"} err="failed to get container status \"fb20c56d24a7d0f90440abcdf9ce95c6e9f3e0f7b3dc4ccab779fcc153276338\": rpc error: code = NotFound desc = could not find container \"fb20c56d24a7d0f90440abcdf9ce95c6e9f3e0f7b3dc4ccab779fcc153276338\": container with ID starting with fb20c56d24a7d0f90440abcdf9ce95c6e9f3e0f7b3dc4ccab779fcc153276338 not found: ID does not exist" Nov 02 23:27:09.621919 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:09.621894 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9da6b802-026a-4118-9833-4802930688d8-kube-api-access-24ksk\" (UniqueName: \"kubernetes.io/projected/9da6b802-026a-4118-9833-4802930688d8-kube-api-access-24ksk\") pod \"9da6b802-026a-4118-9833-4802930688d8\" (UID: \"9da6b802-026a-4118-9833-4802930688d8\") " Nov 02 23:27:09.622072 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:09.621951 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9da6b802-026a-4118-9833-4802930688d8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da6b802-026a-4118-9833-4802930688d8-openshift-service-ca\") pod \"9da6b802-026a-4118-9833-4802930688d8\" (UID: \"9da6b802-026a-4118-9833-4802930688d8\") " Nov 02 23:27:09.622222 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:09.622051 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9da6b802-026a-4118-9833-4802930688d8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9da6b802-026a-4118-9833-4802930688d8-maas-api-tls\") pod \"9da6b802-026a-4118-9833-4802930688d8\" (UID: \"9da6b802-026a-4118-9833-4802930688d8\") " Nov 02 23:27:09.622292 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:09.622271 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9da6b802-026a-4118-9833-4802930688d8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9da6b802-026a-4118-9833-4802930688d8-metrics-certs\") pod \"9da6b802-026a-4118-9833-4802930688d8\" (UID: \"9da6b802-026a-4118-9833-4802930688d8\") " Nov 02 23:27:09.622664 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:09.622626 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9da6b802-026a-4118-9833-4802930688d8-openshift-service-ca" pod "9da6b802-026a-4118-9833-4802930688d8" (UID: "9da6b802-026a-4118-9833-4802930688d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:27:09.622884 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:09.622864 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da6b802-026a-4118-9833-4802930688d8-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:27:09.624779 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:09.624751 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9da6b802-026a-4118-9833-4802930688d8-metrics-certs" pod "9da6b802-026a-4118-9833-4802930688d8" (UID: "9da6b802-026a-4118-9833-4802930688d8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:27:09.625363 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:09.625339 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9da6b802-026a-4118-9833-4802930688d8-kube-api-access-24ksk" pod "9da6b802-026a-4118-9833-4802930688d8" (UID: "9da6b802-026a-4118-9833-4802930688d8"). InnerVolumeSpecName "kube-api-access-24ksk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:27:09.625456 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:09.625404 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9da6b802-026a-4118-9833-4802930688d8-maas-api-tls" pod "9da6b802-026a-4118-9833-4802930688d8" (UID: "9da6b802-026a-4118-9833-4802930688d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:27:09.724024 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:09.723950 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9da6b802-026a-4118-9833-4802930688d8-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:27:09.724024 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:09.723978 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9da6b802-026a-4118-9833-4802930688d8-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:27:09.724024 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:09.723990 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24ksk\" (UniqueName: \"kubernetes.io/projected/9da6b802-026a-4118-9833-4802930688d8-kube-api-access-24ksk\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:27:10.012147 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:10.012047 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf"] Nov 02 23:27:10.017957 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:10.017928 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e875bc-cb4b75c5b-2j4qf"] Nov 02 23:27:11.885372 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:11.885333 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9da6b802-026a-4118-9833-4802930688d8" path="/var/lib/kubelet/pods/9da6b802-026a-4118-9833-4802930688d8/volumes" Nov 02 23:27:27.458267 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:27.458236 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-279zt_f36665e4-38c7-4d93-b047-306185b1a17f/payload-processing/0.log" Nov 02 23:27:27.719612 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:27.719528 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-279zt_f36665e4-38c7-4d93-b047-306185b1a17f/payload-processing/0.log" Nov 02 23:27:27.846535 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:27:27.846489 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:27:29.504517 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:29.504485 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-57f5cf78bc-9ckl7"] Nov 02 23:27:29.505148 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:29.505113 5727 cpu_manager.go:414] "containerMap: removing container" podUID="9da6b802-026a-4118-9833-4802930688d8" containerName="maas-api" Nov 02 23:27:29.505148 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:29.505145 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9da6b802-026a-4118-9833-4802930688d8" containerName="maas-api" Nov 02 23:27:29.505270 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:29.505261 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="9da6b802-026a-4118-9833-4802930688d8" containerName="maas-api" Nov 02 23:27:29.508913 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:29.508896 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-57f5cf78bc-9ckl7" Nov 02 23:27:29.563265 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:29.563168 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-818fe8\"" Nov 02 23:27:29.615936 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:29.615905 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-57f5cf78bc-9ckl7"] Nov 02 23:27:29.663704 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:29.663655 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-818fe8-55ccb9d5b9-p7grv"] Nov 02 23:27:29.666681 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:29.666634 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b4a574-4682-4f13-8883-f55fd0f9d072-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-818fe8-57f5cf78bc-9ckl7\" (UID: \"a1b4a574-4682-4f13-8883-f55fd0f9d072\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-57f5cf78bc-9ckl7" Nov 02 23:27:29.667305 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:29.667280 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kvk4\" (UniqueName: \"kubernetes.io/projected/a1b4a574-4682-4f13-8883-f55fd0f9d072-kube-api-access-8kvk4\") pod \"payload-pre-processing-e2e-ipp-a-818fe8-57f5cf78bc-9ckl7\" (UID: \"a1b4a574-4682-4f13-8883-f55fd0f9d072\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-57f5cf78bc-9ckl7" Nov 02 23:27:29.678613 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:29.678587 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-55ccb9d5b9-p7grv" Nov 02 23:27:29.721562 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:29.721531 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-818fe8-55ccb9d5b9-p7grv"] Nov 02 23:27:29.732226 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:29.732198 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-818fe8-dockercfg-hxd9j\"" Nov 02 23:27:29.769837 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:29.769741 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kvk4\" (UniqueName: \"kubernetes.io/projected/a1b4a574-4682-4f13-8883-f55fd0f9d072-kube-api-access-8kvk4\") pod \"payload-pre-processing-e2e-ipp-a-818fe8-57f5cf78bc-9ckl7\" (UID: \"a1b4a574-4682-4f13-8883-f55fd0f9d072\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-57f5cf78bc-9ckl7" Nov 02 23:27:29.769837 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:29.769817 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b4a574-4682-4f13-8883-f55fd0f9d072-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-818fe8-57f5cf78bc-9ckl7\" (UID: \"a1b4a574-4682-4f13-8883-f55fd0f9d072\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-57f5cf78bc-9ckl7" Nov 02 23:27:29.770068 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:29.769867 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f56a4688-d931-48d4-9b01-eaf8f1232105-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-818fe8-55ccb9d5b9-p7grv\" (UID: \"f56a4688-d931-48d4-9b01-eaf8f1232105\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-55ccb9d5b9-p7grv" Nov 02 23:27:29.770068 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:29.769894 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt5dj\" (UniqueName: \"kubernetes.io/projected/f56a4688-d931-48d4-9b01-eaf8f1232105-kube-api-access-gt5dj\") pod \"payload-processing-e2e-ipp-a-818fe8-55ccb9d5b9-p7grv\" (UID: \"f56a4688-d931-48d4-9b01-eaf8f1232105\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-55ccb9d5b9-p7grv" Nov 02 23:27:29.770767 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:29.770742 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b4a574-4682-4f13-8883-f55fd0f9d072-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-818fe8-57f5cf78bc-9ckl7\" (UID: \"a1b4a574-4682-4f13-8883-f55fd0f9d072\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-57f5cf78bc-9ckl7" Nov 02 23:27:29.830722 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:29.830690 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kvk4\" (UniqueName: \"kubernetes.io/projected/a1b4a574-4682-4f13-8883-f55fd0f9d072-kube-api-access-8kvk4\") pod \"payload-pre-processing-e2e-ipp-a-818fe8-57f5cf78bc-9ckl7\" (UID: \"a1b4a574-4682-4f13-8883-f55fd0f9d072\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-57f5cf78bc-9ckl7" Nov 02 23:27:29.871169 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:29.871108 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f56a4688-d931-48d4-9b01-eaf8f1232105-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-818fe8-55ccb9d5b9-p7grv\" (UID: \"f56a4688-d931-48d4-9b01-eaf8f1232105\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-55ccb9d5b9-p7grv" Nov 02 23:27:29.871341 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:29.871186 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt5dj\" (UniqueName: \"kubernetes.io/projected/f56a4688-d931-48d4-9b01-eaf8f1232105-kube-api-access-gt5dj\") pod \"payload-processing-e2e-ipp-a-818fe8-55ccb9d5b9-p7grv\" (UID: \"f56a4688-d931-48d4-9b01-eaf8f1232105\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-55ccb9d5b9-p7grv" Nov 02 23:27:29.871796 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:29.871774 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f56a4688-d931-48d4-9b01-eaf8f1232105-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-818fe8-55ccb9d5b9-p7grv\" (UID: \"f56a4688-d931-48d4-9b01-eaf8f1232105\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-55ccb9d5b9-p7grv" Nov 02 23:27:29.931877 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:29.931840 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt5dj\" (UniqueName: \"kubernetes.io/projected/f56a4688-d931-48d4-9b01-eaf8f1232105-kube-api-access-gt5dj\") pod \"payload-processing-e2e-ipp-a-818fe8-55ccb9d5b9-p7grv\" (UID: \"f56a4688-d931-48d4-9b01-eaf8f1232105\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-55ccb9d5b9-p7grv" Nov 02 23:27:29.997937 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:29.997903 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-55ccb9d5b9-p7grv" Nov 02 23:27:30.124009 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:30.123976 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-57f5cf78bc-9ckl7" Nov 02 23:27:30.389433 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:30.389404 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-818fe8-55ccb9d5b9-p7grv"] Nov 02 23:27:30.391268 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:27:30.391238 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf56a4688_d931_48d4_9b01_eaf8f1232105.slice/crio-0b45fec4636b4a8d06529316113286782c311d80de513802932043f1f19c069a WatchSource:0}: Error finding container 0b45fec4636b4a8d06529316113286782c311d80de513802932043f1f19c069a: Status 404 returned error can't find the container with id 0b45fec4636b4a8d06529316113286782c311d80de513802932043f1f19c069a Nov 02 23:27:30.519140 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:30.519103 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-57f5cf78bc-9ckl7"] Nov 02 23:27:30.520003 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:27:30.519978 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b4a574_4682_4f13_8883_f55fd0f9d072.slice/crio-acfce5f0ab90964732d7782353e0779956220cdfc0f09174d919cf13dcd0cd35 WatchSource:0}: Error finding container acfce5f0ab90964732d7782353e0779956220cdfc0f09174d919cf13dcd0cd35: Status 404 returned error can't find the container with id acfce5f0ab90964732d7782353e0779956220cdfc0f09174d919cf13dcd0cd35 Nov 02 23:27:30.754629 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:30.753891 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-55ccb9d5b9-p7grv" event={"ID":"f56a4688-d931-48d4-9b01-eaf8f1232105","Type":"ContainerStarted","Data":"79198533f47032475fcf1a34870cee63bacab5f174e3001200f2b89039a02cce"} Nov 02 23:27:30.754629 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:30.753955 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-55ccb9d5b9-p7grv" event={"ID":"f56a4688-d931-48d4-9b01-eaf8f1232105","Type":"ContainerStarted","Data":"0b45fec4636b4a8d06529316113286782c311d80de513802932043f1f19c069a"} Nov 02 23:27:30.754629 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:30.754105 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-55ccb9d5b9-p7grv" Nov 02 23:27:30.755923 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:30.755895 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-57f5cf78bc-9ckl7" event={"ID":"a1b4a574-4682-4f13-8883-f55fd0f9d072","Type":"ContainerStarted","Data":"63f66534616e04ce6f7e4bd417fabee15636cbff0d81e48b2841e5e4abbd0650"} Nov 02 23:27:30.756049 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:30.755945 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-57f5cf78bc-9ckl7" event={"ID":"a1b4a574-4682-4f13-8883-f55fd0f9d072","Type":"ContainerStarted","Data":"acfce5f0ab90964732d7782353e0779956220cdfc0f09174d919cf13dcd0cd35"} Nov 02 23:27:30.756049 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:30.755985 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-57f5cf78bc-9ckl7" Nov 02 23:27:30.870842 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:30.870773 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-55ccb9d5b9-p7grv" podStartSLOduration=1.870754598 podStartE2EDuration="1.870754598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:27:30.868570836 +0000 UTC m=+2337.107068500" watchObservedRunningTime="2026-11-02 23:27:30.870754598 +0000 UTC m=+2337.109252262" Nov 02 23:27:30.986352 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:30.986288 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-57f5cf78bc-9ckl7" podStartSLOduration=1.9862703210000001 podStartE2EDuration="1.986270321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:27:30.982949251 +0000 UTC m=+2337.221446924" watchObservedRunningTime="2026-11-02 23:27:30.986270321 +0000 UTC m=+2337.224767984" Nov 02 23:27:41.778765 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:41.778732 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-57f5cf78bc-9ckl7" Nov 02 23:27:41.779285 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:41.779200 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-55ccb9d5b9-p7grv" Nov 02 23:27:43.501069 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:43.500979 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-adfeab-54956564cf-wqlzh"] Nov 02 23:27:43.506674 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:43.506654 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-adfeab-54956564cf-wqlzh" Nov 02 23:27:43.553022 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:43.552984 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2h8n\" (UniqueName: \"kubernetes.io/projected/89cc96f9-f51b-4132-b2dc-3a926896a29f-kube-api-access-c2h8n\") pod \"payload-pre-processing-e2e-praxis-b-adfeab-54956564cf-wqlzh\" (UID: \"89cc96f9-f51b-4132-b2dc-3a926896a29f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-adfeab-54956564cf-wqlzh" Nov 02 23:27:43.553301 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:43.553277 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89cc96f9-f51b-4132-b2dc-3a926896a29f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-adfeab-54956564cf-wqlzh\" (UID: \"89cc96f9-f51b-4132-b2dc-3a926896a29f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-adfeab-54956564cf-wqlzh" Nov 02 23:27:43.563841 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:43.563813 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-adfeab-54956564cf-wqlzh"] Nov 02 23:27:43.613172 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:43.612439 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-adfeab\"" Nov 02 23:27:43.613172 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:43.612584 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-adfeab-dockercfg-2z9jt\"" Nov 02 23:27:43.654575 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:43.654539 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89cc96f9-f51b-4132-b2dc-3a926896a29f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-adfeab-54956564cf-wqlzh\" (UID: \"89cc96f9-f51b-4132-b2dc-3a926896a29f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-adfeab-54956564cf-wqlzh" Nov 02 23:27:43.654776 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:43.654632 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2h8n\" (UniqueName: \"kubernetes.io/projected/89cc96f9-f51b-4132-b2dc-3a926896a29f-kube-api-access-c2h8n\") pod \"payload-pre-processing-e2e-praxis-b-adfeab-54956564cf-wqlzh\" (UID: \"89cc96f9-f51b-4132-b2dc-3a926896a29f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-adfeab-54956564cf-wqlzh" Nov 02 23:27:43.655573 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:43.655534 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89cc96f9-f51b-4132-b2dc-3a926896a29f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-adfeab-54956564cf-wqlzh\" (UID: \"89cc96f9-f51b-4132-b2dc-3a926896a29f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-adfeab-54956564cf-wqlzh" Nov 02 23:27:43.670632 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:43.670605 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-adfeab-6f58b64699-msrnr"] Nov 02 23:27:43.675759 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:43.675737 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-adfeab-6f58b64699-msrnr" Nov 02 23:27:43.715755 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:43.715729 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2h8n\" (UniqueName: \"kubernetes.io/projected/89cc96f9-f51b-4132-b2dc-3a926896a29f-kube-api-access-c2h8n\") pod \"payload-pre-processing-e2e-praxis-b-adfeab-54956564cf-wqlzh\" (UID: \"89cc96f9-f51b-4132-b2dc-3a926896a29f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-adfeab-54956564cf-wqlzh" Nov 02 23:27:43.730767 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:43.730737 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-adfeab-6f58b64699-msrnr"] Nov 02 23:27:43.755809 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:43.755722 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl8v4\" (UniqueName: \"kubernetes.io/projected/1a7ee0cb-5910-4811-8a00-df5d062ca16c-kube-api-access-pl8v4\") pod \"payload-processing-e2e-praxis-b-adfeab-6f58b64699-msrnr\" (UID: \"1a7ee0cb-5910-4811-8a00-df5d062ca16c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-adfeab-6f58b64699-msrnr" Nov 02 23:27:43.755990 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:43.755964 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7ee0cb-5910-4811-8a00-df5d062ca16c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-adfeab-6f58b64699-msrnr\" (UID: \"1a7ee0cb-5910-4811-8a00-df5d062ca16c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-adfeab-6f58b64699-msrnr" Nov 02 23:27:43.822101 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:43.822063 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-adfeab-54956564cf-wqlzh" Nov 02 23:27:43.857405 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:43.857374 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7ee0cb-5910-4811-8a00-df5d062ca16c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-adfeab-6f58b64699-msrnr\" (UID: \"1a7ee0cb-5910-4811-8a00-df5d062ca16c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-adfeab-6f58b64699-msrnr" Nov 02 23:27:43.857548 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:43.857506 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl8v4\" (UniqueName: \"kubernetes.io/projected/1a7ee0cb-5910-4811-8a00-df5d062ca16c-kube-api-access-pl8v4\") pod \"payload-processing-e2e-praxis-b-adfeab-6f58b64699-msrnr\" (UID: \"1a7ee0cb-5910-4811-8a00-df5d062ca16c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-adfeab-6f58b64699-msrnr" Nov 02 23:27:43.858142 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:43.858108 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7ee0cb-5910-4811-8a00-df5d062ca16c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-adfeab-6f58b64699-msrnr\" (UID: \"1a7ee0cb-5910-4811-8a00-df5d062ca16c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-adfeab-6f58b64699-msrnr" Nov 02 23:27:43.918427 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:43.918392 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl8v4\" (UniqueName: \"kubernetes.io/projected/1a7ee0cb-5910-4811-8a00-df5d062ca16c-kube-api-access-pl8v4\") pod \"payload-processing-e2e-praxis-b-adfeab-6f58b64699-msrnr\" (UID: \"1a7ee0cb-5910-4811-8a00-df5d062ca16c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-adfeab-6f58b64699-msrnr" Nov 02 23:27:43.991539 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:43.991509 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-adfeab-6f58b64699-msrnr" Nov 02 23:27:44.207170 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:44.207112 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-adfeab-54956564cf-wqlzh"] Nov 02 23:27:44.207463 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:27:44.207440 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89cc96f9_f51b_4132_b2dc_3a926896a29f.slice/crio-f6ddf6052f1908b83d567f0adb4eff87ec7244103a5760faba4419af080badac WatchSource:0}: Error finding container f6ddf6052f1908b83d567f0adb4eff87ec7244103a5760faba4419af080badac: Status 404 returned error can't find the container with id f6ddf6052f1908b83d567f0adb4eff87ec7244103a5760faba4419af080badac Nov 02 23:27:44.373661 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:44.373631 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-adfeab-6f58b64699-msrnr"] Nov 02 23:27:44.374411 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:27:44.374389 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a7ee0cb_5910_4811_8a00_df5d062ca16c.slice/crio-b893148f312334a19809a5ad9eacc7ec1557a93415a6039329c53b3db12ea8df WatchSource:0}: Error finding container b893148f312334a19809a5ad9eacc7ec1557a93415a6039329c53b3db12ea8df: Status 404 returned error can't find the container with id b893148f312334a19809a5ad9eacc7ec1557a93415a6039329c53b3db12ea8df Nov 02 23:27:44.885777 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:44.885734 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-adfeab-54956564cf-wqlzh" event={"ID":"89cc96f9-f51b-4132-b2dc-3a926896a29f","Type":"ContainerStarted","Data":"c7fa1d47c78fef51d898ba600b96bceb24b2c67432f23215da696e8daeeb566a"} Nov 02 23:27:44.886277 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:44.885789 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-adfeab-54956564cf-wqlzh" event={"ID":"89cc96f9-f51b-4132-b2dc-3a926896a29f","Type":"ContainerStarted","Data":"f6ddf6052f1908b83d567f0adb4eff87ec7244103a5760faba4419af080badac"} Nov 02 23:27:44.886277 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:44.885823 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-adfeab-54956564cf-wqlzh" Nov 02 23:27:44.887476 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:44.887441 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-adfeab-6f58b64699-msrnr" event={"ID":"1a7ee0cb-5910-4811-8a00-df5d062ca16c","Type":"ContainerStarted","Data":"9b7d5787b990e556d9ccdb22274249881961bc3936911db154dcd459af718b4d"} Nov 02 23:27:44.887587 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:44.887493 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-adfeab-6f58b64699-msrnr" event={"ID":"1a7ee0cb-5910-4811-8a00-df5d062ca16c","Type":"ContainerStarted","Data":"b893148f312334a19809a5ad9eacc7ec1557a93415a6039329c53b3db12ea8df"} Nov 02 23:27:44.887587 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:44.887514 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-adfeab-6f58b64699-msrnr" Nov 02 23:27:45.000683 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:45.000623 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-adfeab-54956564cf-wqlzh" podStartSLOduration=2.000607788 podStartE2EDuration="2.000607788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:27:44.999804466 +0000 UTC m=+2351.238302153" watchObservedRunningTime="2026-11-02 23:27:45.000607788 +0000 UTC m=+2351.239105429" Nov 02 23:27:45.115749 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:45.115685 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-adfeab-6f58b64699-msrnr" podStartSLOduration=2.115666947 podStartE2EDuration="2.115666947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:27:45.114048077 +0000 UTC m=+2351.352545740" watchObservedRunningTime="2026-11-02 23:27:45.115666947 +0000 UTC m=+2351.354164610" Nov 02 23:27:55.897655 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:55.897620 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-adfeab-6f58b64699-msrnr" Nov 02 23:27:55.898197 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:27:55.897872 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-adfeab-54956564cf-wqlzh" Nov 02 23:27:57.846992 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:27:57.846951 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:28:27.845908 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:28:27.845867 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:28:32.748568 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:32.748530 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst"] Nov 02 23:28:32.749001 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:32.748769 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst" podUID="33ee4560-1dc9-4b6c-a3b9-7376b13f5809" containerName="maas-api" containerID="cri-o://4be4a7cd10272fdfabb6d0b4a6767cf78c01fdd8455d081f01d440f6ad2322ba" gracePeriod=30 Nov 02 23:28:33.053777 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:33.053748 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst" Nov 02 23:28:33.164706 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:33.164642 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33ee4560-1dc9-4b6c-a3b9-7376b13f5809-metrics-certs\" (UniqueName: \"kubernetes.io/secret/33ee4560-1dc9-4b6c-a3b9-7376b13f5809-metrics-certs\") pod \"33ee4560-1dc9-4b6c-a3b9-7376b13f5809\" (UID: \"33ee4560-1dc9-4b6c-a3b9-7376b13f5809\") " Nov 02 23:28:33.164706 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:33.164719 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33ee4560-1dc9-4b6c-a3b9-7376b13f5809-kube-api-access-fxfkl\" (UniqueName: \"kubernetes.io/projected/33ee4560-1dc9-4b6c-a3b9-7376b13f5809-kube-api-access-fxfkl\") pod \"33ee4560-1dc9-4b6c-a3b9-7376b13f5809\" (UID: \"33ee4560-1dc9-4b6c-a3b9-7376b13f5809\") " Nov 02 23:28:33.164954 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:33.164823 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33ee4560-1dc9-4b6c-a3b9-7376b13f5809-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33ee4560-1dc9-4b6c-a3b9-7376b13f5809-maas-api-tls\") pod \"33ee4560-1dc9-4b6c-a3b9-7376b13f5809\" (UID: \"33ee4560-1dc9-4b6c-a3b9-7376b13f5809\") " Nov 02 23:28:33.164954 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:33.164915 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33ee4560-1dc9-4b6c-a3b9-7376b13f5809-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33ee4560-1dc9-4b6c-a3b9-7376b13f5809-openshift-service-ca\") pod \"33ee4560-1dc9-4b6c-a3b9-7376b13f5809\" (UID: \"33ee4560-1dc9-4b6c-a3b9-7376b13f5809\") " Nov 02 23:28:33.165319 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:33.165287 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ee4560-1dc9-4b6c-a3b9-7376b13f5809-openshift-service-ca" pod "33ee4560-1dc9-4b6c-a3b9-7376b13f5809" (UID: "33ee4560-1dc9-4b6c-a3b9-7376b13f5809"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:28:33.165576 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:33.165556 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33ee4560-1dc9-4b6c-a3b9-7376b13f5809-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:33.167218 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:33.167104 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33ee4560-1dc9-4b6c-a3b9-7376b13f5809-maas-api-tls" pod "33ee4560-1dc9-4b6c-a3b9-7376b13f5809" (UID: "33ee4560-1dc9-4b6c-a3b9-7376b13f5809"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:28:33.167218 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:33.167178 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33ee4560-1dc9-4b6c-a3b9-7376b13f5809-metrics-certs" pod "33ee4560-1dc9-4b6c-a3b9-7376b13f5809" (UID: "33ee4560-1dc9-4b6c-a3b9-7376b13f5809"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:28:33.167592 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:33.167576 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ee4560-1dc9-4b6c-a3b9-7376b13f5809-kube-api-access-fxfkl" pod "33ee4560-1dc9-4b6c-a3b9-7376b13f5809" (UID: "33ee4560-1dc9-4b6c-a3b9-7376b13f5809"). InnerVolumeSpecName "kube-api-access-fxfkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:33.266954 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:33.266920 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33ee4560-1dc9-4b6c-a3b9-7376b13f5809-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:33.266954 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:33.266950 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxfkl\" (UniqueName: \"kubernetes.io/projected/33ee4560-1dc9-4b6c-a3b9-7376b13f5809-kube-api-access-fxfkl\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:33.267233 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:33.266966 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33ee4560-1dc9-4b6c-a3b9-7376b13f5809-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:33.268698 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:33.268627 5727 generic.go:356] "Generic (PLEG): container finished" podID="33ee4560-1dc9-4b6c-a3b9-7376b13f5809" containerID="4be4a7cd10272fdfabb6d0b4a6767cf78c01fdd8455d081f01d440f6ad2322ba" exitCode=0 Nov 02 23:28:33.268698 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:33.268683 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst" Nov 02 23:28:33.268859 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:33.268716 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst" event={"ID":"33ee4560-1dc9-4b6c-a3b9-7376b13f5809","Type":"ContainerDied","Data":"4be4a7cd10272fdfabb6d0b4a6767cf78c01fdd8455d081f01d440f6ad2322ba"} Nov 02 23:28:33.268859 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:33.268771 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst" event={"ID":"33ee4560-1dc9-4b6c-a3b9-7376b13f5809","Type":"ContainerDied","Data":"8c502906304d5e826de5b038323f172c239dd4c5bb427f99bb7516320bf4b1e3"} Nov 02 23:28:33.268859 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:33.268794 5727 scope.go:122] "RemoveContainer" containerID="4be4a7cd10272fdfabb6d0b4a6767cf78c01fdd8455d081f01d440f6ad2322ba" Nov 02 23:28:33.280453 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:33.280434 5727 scope.go:122] "RemoveContainer" containerID="4be4a7cd10272fdfabb6d0b4a6767cf78c01fdd8455d081f01d440f6ad2322ba" Nov 02 23:28:33.280739 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:28:33.280721 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4be4a7cd10272fdfabb6d0b4a6767cf78c01fdd8455d081f01d440f6ad2322ba\": container with ID starting with 4be4a7cd10272fdfabb6d0b4a6767cf78c01fdd8455d081f01d440f6ad2322ba not found: ID does not exist" containerID="4be4a7cd10272fdfabb6d0b4a6767cf78c01fdd8455d081f01d440f6ad2322ba" Nov 02 23:28:33.280786 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:33.280748 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4be4a7cd10272fdfabb6d0b4a6767cf78c01fdd8455d081f01d440f6ad2322ba"} err="failed to get container status \"4be4a7cd10272fdfabb6d0b4a6767cf78c01fdd8455d081f01d440f6ad2322ba\": rpc error: code = NotFound desc = could not find container \"4be4a7cd10272fdfabb6d0b4a6767cf78c01fdd8455d081f01d440f6ad2322ba\": container with ID starting with 4be4a7cd10272fdfabb6d0b4a6767cf78c01fdd8455d081f01d440f6ad2322ba not found: ID does not exist" Nov 02 23:28:33.433772 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:33.433733 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst"] Nov 02 23:28:33.437564 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:33.437534 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-03fe6e-54f7fb859-xgpst"] Nov 02 23:28:33.883845 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:33.883806 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33ee4560-1dc9-4b6c-a3b9-7376b13f5809" path="/var/lib/kubelet/pods/33ee4560-1dc9-4b6c-a3b9-7376b13f5809/volumes" Nov 02 23:28:34.484794 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:34.484767 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 02 23:28:34.500477 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:34.500450 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 23:28:34.562953 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:34.562926 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 02 23:28:34.578265 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:34.578240 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 23:28:52.233035 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:52.232997 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a2f549-7c54bf87f5-6g6xt"] Nov 02 23:28:52.339746 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:52.339699 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f549-68b787d455-pcjmj"] Nov 02 23:28:57.234055 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:57.234010 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f549-7c54bf87f5-6g6xt" podUID="629bc7d8-4ece-473f-98b0-22a2571cd5e5" containerName="payload-processing" containerID="cri-o://be4afe45ccb9a0c55d449b79968e0b39c7d1aa60852949800adcb0a095bb4862" gracePeriod=25 Nov 02 23:28:57.340651 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:57.340610 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f549-68b787d455-pcjmj" podUID="9c56d8a2-f496-4fd2-acf6-8f27122bddae" containerName="payload-pre-processing" containerID="cri-o://7fc3e5e9aab41fde9fd2167e7fb59cbcd81a5297d7bcaa828c672f0f58f16bcf" gracePeriod=25 Nov 02 23:28:57.511167 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:57.509811 5727 generic.go:356] "Generic (PLEG): container finished" podID="629bc7d8-4ece-473f-98b0-22a2571cd5e5" containerID="be4afe45ccb9a0c55d449b79968e0b39c7d1aa60852949800adcb0a095bb4862" exitCode=0 Nov 02 23:28:57.511167 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:57.509937 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f549-7c54bf87f5-6g6xt" event={"ID":"629bc7d8-4ece-473f-98b0-22a2571cd5e5","Type":"ContainerDied","Data":"be4afe45ccb9a0c55d449b79968e0b39c7d1aa60852949800adcb0a095bb4862"} Nov 02 23:28:57.511855 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:57.511830 5727 generic.go:356] "Generic (PLEG): container finished" podID="9c56d8a2-f496-4fd2-acf6-8f27122bddae" containerID="7fc3e5e9aab41fde9fd2167e7fb59cbcd81a5297d7bcaa828c672f0f58f16bcf" exitCode=0 Nov 02 23:28:57.511918 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:57.511901 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f549-68b787d455-pcjmj" event={"ID":"9c56d8a2-f496-4fd2-acf6-8f27122bddae","Type":"ContainerDied","Data":"7fc3e5e9aab41fde9fd2167e7fb59cbcd81a5297d7bcaa828c672f0f58f16bcf"} Nov 02 23:28:57.541904 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:57.541870 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f549-7c54bf87f5-6g6xt" Nov 02 23:28:57.578564 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:57.578530 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/629bc7d8-4ece-473f-98b0-22a2571cd5e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/629bc7d8-4ece-473f-98b0-22a2571cd5e5-plugins-config-volume\") pod \"629bc7d8-4ece-473f-98b0-22a2571cd5e5\" (UID: \"629bc7d8-4ece-473f-98b0-22a2571cd5e5\") " Nov 02 23:28:57.578741 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:57.578669 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/629bc7d8-4ece-473f-98b0-22a2571cd5e5-kube-api-access-vbv92\" (UniqueName: \"kubernetes.io/projected/629bc7d8-4ece-473f-98b0-22a2571cd5e5-kube-api-access-vbv92\") pod \"629bc7d8-4ece-473f-98b0-22a2571cd5e5\" (UID: \"629bc7d8-4ece-473f-98b0-22a2571cd5e5\") " Nov 02 23:28:57.578997 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:57.578969 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/629bc7d8-4ece-473f-98b0-22a2571cd5e5-plugins-config-volume" pod "629bc7d8-4ece-473f-98b0-22a2571cd5e5" (UID: "629bc7d8-4ece-473f-98b0-22a2571cd5e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:28:57.579277 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:57.579252 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/629bc7d8-4ece-473f-98b0-22a2571cd5e5-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:57.583605 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:57.583566 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629bc7d8-4ece-473f-98b0-22a2571cd5e5-kube-api-access-vbv92" pod "629bc7d8-4ece-473f-98b0-22a2571cd5e5" (UID: "629bc7d8-4ece-473f-98b0-22a2571cd5e5"). InnerVolumeSpecName "kube-api-access-vbv92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:57.641033 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:57.641004 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f549-68b787d455-pcjmj" Nov 02 23:28:57.680660 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:57.680564 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c56d8a2-f496-4fd2-acf6-8f27122bddae-kube-api-access-9d2tb\" (UniqueName: \"kubernetes.io/projected/9c56d8a2-f496-4fd2-acf6-8f27122bddae-kube-api-access-9d2tb\") pod \"9c56d8a2-f496-4fd2-acf6-8f27122bddae\" (UID: \"9c56d8a2-f496-4fd2-acf6-8f27122bddae\") " Nov 02 23:28:57.680660 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:57.680622 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c56d8a2-f496-4fd2-acf6-8f27122bddae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c56d8a2-f496-4fd2-acf6-8f27122bddae-plugins-config-volume\") pod \"9c56d8a2-f496-4fd2-acf6-8f27122bddae\" (UID: \"9c56d8a2-f496-4fd2-acf6-8f27122bddae\") " Nov 02 23:28:57.681229 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:57.681208 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbv92\" (UniqueName: \"kubernetes.io/projected/629bc7d8-4ece-473f-98b0-22a2571cd5e5-kube-api-access-vbv92\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:57.681585 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:57.681556 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c56d8a2-f496-4fd2-acf6-8f27122bddae-plugins-config-volume" pod "9c56d8a2-f496-4fd2-acf6-8f27122bddae" (UID: "9c56d8a2-f496-4fd2-acf6-8f27122bddae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:28:57.684111 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:57.684084 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c56d8a2-f496-4fd2-acf6-8f27122bddae-kube-api-access-9d2tb" pod "9c56d8a2-f496-4fd2-acf6-8f27122bddae" (UID: "9c56d8a2-f496-4fd2-acf6-8f27122bddae"). InnerVolumeSpecName "kube-api-access-9d2tb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:57.782651 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:57.782613 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d2tb\" (UniqueName: \"kubernetes.io/projected/9c56d8a2-f496-4fd2-acf6-8f27122bddae-kube-api-access-9d2tb\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:57.782651 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:57.782654 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c56d8a2-f496-4fd2-acf6-8f27122bddae-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:57.846210 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:28:57.846170 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:28:58.520935 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:58.520858 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f549-7c54bf87f5-6g6xt" Nov 02 23:28:58.520935 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:58.520865 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f549-7c54bf87f5-6g6xt" event={"ID":"629bc7d8-4ece-473f-98b0-22a2571cd5e5","Type":"ContainerDied","Data":"a7653cc558e458bd97aa3bf97289d9b9652211359f4b93dfe3513e3eb32fd536"} Nov 02 23:28:58.521475 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:58.520936 5727 scope.go:122] "RemoveContainer" containerID="be4afe45ccb9a0c55d449b79968e0b39c7d1aa60852949800adcb0a095bb4862" Nov 02 23:28:58.523187 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:58.522631 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f549-68b787d455-pcjmj" Nov 02 23:28:58.523187 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:58.522642 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f549-68b787d455-pcjmj" event={"ID":"9c56d8a2-f496-4fd2-acf6-8f27122bddae","Type":"ContainerDied","Data":"7a1428c5ce7de48411b99266eeabb6a2829d6fe3e337c46b76b98190f9afca4e"} Nov 02 23:28:58.532398 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:58.532378 5727 scope.go:122] "RemoveContainer" containerID="7fc3e5e9aab41fde9fd2167e7fb59cbcd81a5297d7bcaa828c672f0f58f16bcf" Nov 02 23:28:58.688184 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:58.688146 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a2f549-7c54bf87f5-6g6xt"] Nov 02 23:28:58.735385 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:58.735353 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a2f549-7c54bf87f5-6g6xt"] Nov 02 23:28:58.799981 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:58.799905 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f549-68b787d455-pcjmj"] Nov 02 23:28:58.805561 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:58.805532 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f549-68b787d455-pcjmj"] Nov 02 23:28:59.896407 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:59.896365 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="629bc7d8-4ece-473f-98b0-22a2571cd5e5" path="/var/lib/kubelet/pods/629bc7d8-4ece-473f-98b0-22a2571cd5e5/volumes" Nov 02 23:28:59.896962 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:28:59.896944 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c56d8a2-f496-4fd2-acf6-8f27122bddae" path="/var/lib/kubelet/pods/9c56d8a2-f496-4fd2-acf6-8f27122bddae/volumes" Nov 02 23:29:21.208882 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:21.208801 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-85efc6-74998f459-r4dgk"] Nov 02 23:29:21.315894 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:21.315856 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-85efc6-7dd5465b9f-t426x"] Nov 02 23:29:26.317745 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:26.317695 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85efc6-7dd5465b9f-t426x" podUID="c559df9f-192e-47d0-8b88-e13ff6259fd6" containerName="payload-pre-processing" containerID="cri-o://bc0419cb75b2ce8265ca3131615650ac4ee8cb45bef6fc9e8be26a5afee5f42e" gracePeriod=25 Nov 02 23:29:26.606519 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:26.606489 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85efc6-7dd5465b9f-t426x" Nov 02 23:29:26.665481 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:26.665453 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c559df9f-192e-47d0-8b88-e13ff6259fd6-kube-api-access-jdwcx\" (UniqueName: \"kubernetes.io/projected/c559df9f-192e-47d0-8b88-e13ff6259fd6-kube-api-access-jdwcx\") pod \"c559df9f-192e-47d0-8b88-e13ff6259fd6\" (UID: \"c559df9f-192e-47d0-8b88-e13ff6259fd6\") " Nov 02 23:29:26.665655 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:26.665628 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c559df9f-192e-47d0-8b88-e13ff6259fd6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c559df9f-192e-47d0-8b88-e13ff6259fd6-plugins-config-volume\") pod \"c559df9f-192e-47d0-8b88-e13ff6259fd6\" (UID: \"c559df9f-192e-47d0-8b88-e13ff6259fd6\") " Nov 02 23:29:26.666164 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:26.666059 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c559df9f-192e-47d0-8b88-e13ff6259fd6-plugins-config-volume" pod "c559df9f-192e-47d0-8b88-e13ff6259fd6" (UID: "c559df9f-192e-47d0-8b88-e13ff6259fd6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:29:26.667841 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:26.667809 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c559df9f-192e-47d0-8b88-e13ff6259fd6-kube-api-access-jdwcx" pod "c559df9f-192e-47d0-8b88-e13ff6259fd6" (UID: "c559df9f-192e-47d0-8b88-e13ff6259fd6"). InnerVolumeSpecName "kube-api-access-jdwcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:29:26.704274 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:26.704241 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-85efc6-74998f459-r4dgk" podUID="c53de2f2-1902-4cd3-94d6-2f06ab70aa69" containerName="payload-processing" containerID="cri-o://63af311d74a2d0ae05ed342a326aa70173a27cda7118e0d0291514c0c8a237cc" gracePeriod=25 Nov 02 23:29:26.743721 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:26.743683 5727 generic.go:356] "Generic (PLEG): container finished" podID="c559df9f-192e-47d0-8b88-e13ff6259fd6" containerID="bc0419cb75b2ce8265ca3131615650ac4ee8cb45bef6fc9e8be26a5afee5f42e" exitCode=0 Nov 02 23:29:26.743875 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:26.743744 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85efc6-7dd5465b9f-t426x" Nov 02 23:29:26.743875 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:26.743735 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85efc6-7dd5465b9f-t426x" event={"ID":"c559df9f-192e-47d0-8b88-e13ff6259fd6","Type":"ContainerDied","Data":"bc0419cb75b2ce8265ca3131615650ac4ee8cb45bef6fc9e8be26a5afee5f42e"} Nov 02 23:29:26.743982 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:26.743898 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85efc6-7dd5465b9f-t426x" event={"ID":"c559df9f-192e-47d0-8b88-e13ff6259fd6","Type":"ContainerDied","Data":"731ced98204f520efb83d8fe4030c007065d6545c62d8e13b8f9c0db2d623f00"} Nov 02 23:29:26.743982 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:26.743931 5727 scope.go:122] "RemoveContainer" containerID="bc0419cb75b2ce8265ca3131615650ac4ee8cb45bef6fc9e8be26a5afee5f42e" Nov 02 23:29:26.767029 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:26.767000 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c559df9f-192e-47d0-8b88-e13ff6259fd6-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:26.767029 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:26.767030 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdwcx\" (UniqueName: \"kubernetes.io/projected/c559df9f-192e-47d0-8b88-e13ff6259fd6-kube-api-access-jdwcx\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:26.822396 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:26.822370 5727 scope.go:122] "RemoveContainer" containerID="bc0419cb75b2ce8265ca3131615650ac4ee8cb45bef6fc9e8be26a5afee5f42e" Nov 02 23:29:26.822804 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:29:26.822771 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc0419cb75b2ce8265ca3131615650ac4ee8cb45bef6fc9e8be26a5afee5f42e\": container with ID starting with bc0419cb75b2ce8265ca3131615650ac4ee8cb45bef6fc9e8be26a5afee5f42e not found: ID does not exist" containerID="bc0419cb75b2ce8265ca3131615650ac4ee8cb45bef6fc9e8be26a5afee5f42e" Nov 02 23:29:26.822926 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:26.822814 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc0419cb75b2ce8265ca3131615650ac4ee8cb45bef6fc9e8be26a5afee5f42e"} err="failed to get container status \"bc0419cb75b2ce8265ca3131615650ac4ee8cb45bef6fc9e8be26a5afee5f42e\": rpc error: code = NotFound desc = could not find container \"bc0419cb75b2ce8265ca3131615650ac4ee8cb45bef6fc9e8be26a5afee5f42e\": container with ID starting with bc0419cb75b2ce8265ca3131615650ac4ee8cb45bef6fc9e8be26a5afee5f42e not found: ID does not exist" Nov 02 23:29:26.910842 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:26.910809 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-85efc6-7dd5465b9f-t426x"] Nov 02 23:29:26.914092 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:26.914063 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-85efc6-7dd5465b9f-t426x"] Nov 02 23:29:26.961861 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:26.961837 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-85efc6-74998f459-r4dgk" Nov 02 23:29:27.070976 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:27.070930 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c53de2f2-1902-4cd3-94d6-2f06ab70aa69-kube-api-access-pnktz\" (UniqueName: \"kubernetes.io/projected/c53de2f2-1902-4cd3-94d6-2f06ab70aa69-kube-api-access-pnktz\") pod \"c53de2f2-1902-4cd3-94d6-2f06ab70aa69\" (UID: \"c53de2f2-1902-4cd3-94d6-2f06ab70aa69\") " Nov 02 23:29:27.071177 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:27.071098 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c53de2f2-1902-4cd3-94d6-2f06ab70aa69-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c53de2f2-1902-4cd3-94d6-2f06ab70aa69-plugins-config-volume\") pod \"c53de2f2-1902-4cd3-94d6-2f06ab70aa69\" (UID: \"c53de2f2-1902-4cd3-94d6-2f06ab70aa69\") " Nov 02 23:29:27.071502 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:27.071473 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c53de2f2-1902-4cd3-94d6-2f06ab70aa69-plugins-config-volume" pod "c53de2f2-1902-4cd3-94d6-2f06ab70aa69" (UID: "c53de2f2-1902-4cd3-94d6-2f06ab70aa69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:29:27.071664 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:27.071649 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c53de2f2-1902-4cd3-94d6-2f06ab70aa69-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:27.073426 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:27.073403 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53de2f2-1902-4cd3-94d6-2f06ab70aa69-kube-api-access-pnktz" pod "c53de2f2-1902-4cd3-94d6-2f06ab70aa69" (UID: "c53de2f2-1902-4cd3-94d6-2f06ab70aa69"). InnerVolumeSpecName "kube-api-access-pnktz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:29:27.172585 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:27.172507 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnktz\" (UniqueName: \"kubernetes.io/projected/c53de2f2-1902-4cd3-94d6-2f06ab70aa69-kube-api-access-pnktz\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:27.757408 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:27.757367 5727 generic.go:356] "Generic (PLEG): container finished" podID="c53de2f2-1902-4cd3-94d6-2f06ab70aa69" containerID="63af311d74a2d0ae05ed342a326aa70173a27cda7118e0d0291514c0c8a237cc" exitCode=0 Nov 02 23:29:27.757919 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:27.757428 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-85efc6-74998f459-r4dgk" Nov 02 23:29:27.757919 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:27.757423 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-85efc6-74998f459-r4dgk" event={"ID":"c53de2f2-1902-4cd3-94d6-2f06ab70aa69","Type":"ContainerDied","Data":"63af311d74a2d0ae05ed342a326aa70173a27cda7118e0d0291514c0c8a237cc"} Nov 02 23:29:27.757919 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:27.757579 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-85efc6-74998f459-r4dgk" event={"ID":"c53de2f2-1902-4cd3-94d6-2f06ab70aa69","Type":"ContainerDied","Data":"5c650cbeeff1f509027d483393ba7abdc40129a0924308124f29eebf10dfe107"} Nov 02 23:29:27.757919 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:27.757603 5727 scope.go:122] "RemoveContainer" containerID="63af311d74a2d0ae05ed342a326aa70173a27cda7118e0d0291514c0c8a237cc" Nov 02 23:29:27.768627 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:27.768608 5727 scope.go:122] "RemoveContainer" containerID="63af311d74a2d0ae05ed342a326aa70173a27cda7118e0d0291514c0c8a237cc" Nov 02 23:29:27.768894 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:29:27.768871 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63af311d74a2d0ae05ed342a326aa70173a27cda7118e0d0291514c0c8a237cc\": container with ID starting with 63af311d74a2d0ae05ed342a326aa70173a27cda7118e0d0291514c0c8a237cc not found: ID does not exist" containerID="63af311d74a2d0ae05ed342a326aa70173a27cda7118e0d0291514c0c8a237cc" Nov 02 23:29:27.768945 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:27.768905 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63af311d74a2d0ae05ed342a326aa70173a27cda7118e0d0291514c0c8a237cc"} err="failed to get container status \"63af311d74a2d0ae05ed342a326aa70173a27cda7118e0d0291514c0c8a237cc\": rpc error: code = NotFound desc = could not find container \"63af311d74a2d0ae05ed342a326aa70173a27cda7118e0d0291514c0c8a237cc\": container with ID starting with 63af311d74a2d0ae05ed342a326aa70173a27cda7118e0d0291514c0c8a237cc not found: ID does not exist" Nov 02 23:29:27.846679 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:29:27.846639 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:29:27.883680 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:27.883644 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c559df9f-192e-47d0-8b88-e13ff6259fd6" path="/var/lib/kubelet/pods/c559df9f-192e-47d0-8b88-e13ff6259fd6/volumes" Nov 02 23:29:27.904971 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:27.904940 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-b-85efc6-74998f459-r4dgk" podUID="c53de2f2-1902-4cd3-94d6-2f06ab70aa69" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.134:50052\" within 1s: context deadline exceeded" Nov 02 23:29:27.905156 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:29:27.905019 5727 logging.go:55] [core] [Channel #5192 SubChannel #5193]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.134:50052", ServerName: "10.132.0.134:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.134:50052: operation was canceled" Nov 02 23:29:27.923713 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:27.923685 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-85efc6-74998f459-r4dgk"] Nov 02 23:29:27.927456 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:27.927429 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-85efc6-74998f459-r4dgk"] Nov 02 23:29:29.884226 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:29.884192 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c53de2f2-1902-4cd3-94d6-2f06ab70aa69" path="/var/lib/kubelet/pods/c53de2f2-1902-4cd3-94d6-2f06ab70aa69/volumes" Nov 02 23:29:49.340189 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:49.340155 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-818fe8-55ccb9d5b9-p7grv"] Nov 02 23:29:49.450581 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:49.446798 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-57f5cf78bc-9ckl7"] Nov 02 23:29:52.419089 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.419056 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj"] Nov 02 23:29:52.419710 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.419695 5727 cpu_manager.go:414] "containerMap: removing container" podUID="c53de2f2-1902-4cd3-94d6-2f06ab70aa69" containerName="payload-processing" Nov 02 23:29:52.419755 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.419712 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c53de2f2-1902-4cd3-94d6-2f06ab70aa69" containerName="payload-processing" Nov 02 23:29:52.419755 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.419735 5727 cpu_manager.go:414] "containerMap: removing container" podUID="9c56d8a2-f496-4fd2-acf6-8f27122bddae" containerName="payload-pre-processing" Nov 02 23:29:52.419755 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.419740 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c56d8a2-f496-4fd2-acf6-8f27122bddae" containerName="payload-pre-processing" Nov 02 23:29:52.419755 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.419754 5727 cpu_manager.go:414] "containerMap: removing container" podUID="c559df9f-192e-47d0-8b88-e13ff6259fd6" containerName="payload-pre-processing" Nov 02 23:29:52.419905 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.419759 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c559df9f-192e-47d0-8b88-e13ff6259fd6" containerName="payload-pre-processing" Nov 02 23:29:52.419905 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.419778 5727 cpu_manager.go:414] "containerMap: removing container" podUID="629bc7d8-4ece-473f-98b0-22a2571cd5e5" containerName="payload-processing" Nov 02 23:29:52.419905 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.419782 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="629bc7d8-4ece-473f-98b0-22a2571cd5e5" containerName="payload-processing" Nov 02 23:29:52.419905 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.419789 5727 cpu_manager.go:414] "containerMap: removing container" podUID="33ee4560-1dc9-4b6c-a3b9-7376b13f5809" containerName="maas-api" Nov 02 23:29:52.419905 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.419793 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33ee4560-1dc9-4b6c-a3b9-7376b13f5809" containerName="maas-api" Nov 02 23:29:52.419905 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.419892 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="c53de2f2-1902-4cd3-94d6-2f06ab70aa69" containerName="payload-processing" Nov 02 23:29:52.419905 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.419904 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c56d8a2-f496-4fd2-acf6-8f27122bddae" containerName="payload-pre-processing" Nov 02 23:29:52.420106 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.419912 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="c559df9f-192e-47d0-8b88-e13ff6259fd6" containerName="payload-pre-processing" Nov 02 23:29:52.420106 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.419920 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="629bc7d8-4ece-473f-98b0-22a2571cd5e5" containerName="payload-processing" Nov 02 23:29:52.420106 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.419925 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="33ee4560-1dc9-4b6c-a3b9-7376b13f5809" containerName="maas-api" Nov 02 23:29:52.425024 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.425007 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:29:52.476839 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.476807 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-818fe8-openshift-default-dockercfg-j2vfc\"" Nov 02 23:29:52.490610 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.490568 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istio-token\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:29:52.490772 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.490706 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-credential-socket\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:29:52.490772 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.490762 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istiod-ca-cert\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:29:52.490898 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.490880 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istio-podinfo\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:29:52.490961 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.490913 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dxj7\" (UniqueName: \"kubernetes.io/projected/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-kube-api-access-6dxj7\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:29:52.491017 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.490997 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-workload-certs\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:29:52.491071 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.491049 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istio-envoy\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:29:52.491154 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.491083 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istio-data\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:29:52.491154 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.491137 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-workload-socket\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:29:52.592346 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.592302 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-workload-socket\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:29:52.592540 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.592411 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istio-token\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:29:52.592540 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.592508 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-credential-socket\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:29:52.592671 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.592567 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istiod-ca-cert\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:29:52.592671 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.592637 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istio-podinfo\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:29:52.592671 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.592665 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dxj7\" (UniqueName: \"kubernetes.io/projected/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-kube-api-access-6dxj7\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:29:52.592826 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.592712 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-workload-certs\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:29:52.592826 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.592750 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-workload-socket\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:29:52.592826 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.592786 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istio-envoy\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:29:52.592826 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.592817 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istio-data\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:29:52.593242 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.593209 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istio-data\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:29:52.593378 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.593355 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-workload-certs\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:29:52.593542 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.593507 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-credential-socket\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:29:52.593952 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.593932 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istiod-ca-cert\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:29:52.595939 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.595901 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istio-podinfo\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:29:52.596685 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.596423 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istio-envoy\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:29:52.631600 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.631566 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj"] Nov 02 23:29:52.660152 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.660073 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istio-token\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:29:52.669628 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.668656 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dxj7\" (UniqueName: \"kubernetes.io/projected/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-kube-api-access-6dxj7\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:29:52.739177 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:52.739114 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:29:53.180573 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:53.180537 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj"] Nov 02 23:29:53.220295 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:29:53.220254 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c0fb24_5f82_4c08_bc9b_f80476f4ded9.slice/crio-e470cbb49c5f76e9063fa5ff41177c4a8632c7f87417a0b72b234fc414667d92 WatchSource:0}: Error finding container e470cbb49c5f76e9063fa5ff41177c4a8632c7f87417a0b72b234fc414667d92: Status 404 returned error can't find the container with id e470cbb49c5f76e9063fa5ff41177c4a8632c7f87417a0b72b234fc414667d92 Nov 02 23:29:53.223105 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:53.223065 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:29:53.223210 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:53.223185 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:29:53.223251 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:53.223230 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:29:53.984540 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:53.984499 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" event={"ID":"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9","Type":"ContainerStarted","Data":"f15ea955500c4e2237ee2c82e3d19c7074202d51608e7928de28df1ba4ff0cd5"} Nov 02 23:29:53.985008 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:53.984561 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" event={"ID":"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9","Type":"ContainerStarted","Data":"e470cbb49c5f76e9063fa5ff41177c4a8632c7f87417a0b72b234fc414667d92"} Nov 02 23:29:54.191991 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:54.191917 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" podStartSLOduration=2.191894648 podStartE2EDuration="2.191894648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:29:54.162634613 +0000 UTC m=+2480.401132329" watchObservedRunningTime="2026-11-02 23:29:54.191894648 +0000 UTC m=+2480.430392311" Nov 02 23:29:54.341377 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:54.341329 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-55ccb9d5b9-p7grv" podUID="f56a4688-d931-48d4-9b01-eaf8f1232105" containerName="payload-processing" containerID="cri-o://79198533f47032475fcf1a34870cee63bacab5f174e3001200f2b89039a02cce" gracePeriod=25 Nov 02 23:29:54.448147 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:54.448041 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-57f5cf78bc-9ckl7" podUID="a1b4a574-4682-4f13-8883-f55fd0f9d072" containerName="payload-pre-processing" containerID="cri-o://63f66534616e04ce6f7e4bd417fabee15636cbff0d81e48b2841e5e4abbd0650" gracePeriod=25 Nov 02 23:29:54.771154 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:54.762875 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:29:54.790322 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:54.790281 5727 patch_prober.go:29] interesting pod/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 02 23:29:54.790526 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:54.790393 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" podUID="c5c0fb24-5f82-4c08-bc9b-f80476f4ded9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 02 23:29:54.791209 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:54.791181 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-55ccb9d5b9-p7grv" Nov 02 23:29:54.824029 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:54.823991 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f56a4688-d931-48d4-9b01-eaf8f1232105-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f56a4688-d931-48d4-9b01-eaf8f1232105-plugins-config-volume\") pod \"f56a4688-d931-48d4-9b01-eaf8f1232105\" (UID: \"f56a4688-d931-48d4-9b01-eaf8f1232105\") " Nov 02 23:29:54.824250 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:54.824068 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f56a4688-d931-48d4-9b01-eaf8f1232105-kube-api-access-gt5dj\" (UniqueName: \"kubernetes.io/projected/f56a4688-d931-48d4-9b01-eaf8f1232105-kube-api-access-gt5dj\") pod \"f56a4688-d931-48d4-9b01-eaf8f1232105\" (UID: \"f56a4688-d931-48d4-9b01-eaf8f1232105\") " Nov 02 23:29:54.825262 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:54.825231 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f56a4688-d931-48d4-9b01-eaf8f1232105-plugins-config-volume" pod "f56a4688-d931-48d4-9b01-eaf8f1232105" (UID: "f56a4688-d931-48d4-9b01-eaf8f1232105"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:29:54.830872 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:54.830839 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f56a4688-d931-48d4-9b01-eaf8f1232105-kube-api-access-gt5dj" pod "f56a4688-d931-48d4-9b01-eaf8f1232105" (UID: "f56a4688-d931-48d4-9b01-eaf8f1232105"). InnerVolumeSpecName "kube-api-access-gt5dj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:29:54.896335 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:54.896308 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-57f5cf78bc-9ckl7" Nov 02 23:29:54.925892 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:54.925862 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1b4a574-4682-4f13-8883-f55fd0f9d072-kube-api-access-8kvk4\" (UniqueName: \"kubernetes.io/projected/a1b4a574-4682-4f13-8883-f55fd0f9d072-kube-api-access-8kvk4\") pod \"a1b4a574-4682-4f13-8883-f55fd0f9d072\" (UID: \"a1b4a574-4682-4f13-8883-f55fd0f9d072\") " Nov 02 23:29:54.926161 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:54.926147 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1b4a574-4682-4f13-8883-f55fd0f9d072-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b4a574-4682-4f13-8883-f55fd0f9d072-plugins-config-volume\") pod \"a1b4a574-4682-4f13-8883-f55fd0f9d072\" (UID: \"a1b4a574-4682-4f13-8883-f55fd0f9d072\") " Nov 02 23:29:54.926483 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:54.926459 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1b4a574-4682-4f13-8883-f55fd0f9d072-plugins-config-volume" pod "a1b4a574-4682-4f13-8883-f55fd0f9d072" (UID: "a1b4a574-4682-4f13-8883-f55fd0f9d072"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:29:54.926725 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:54.926711 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f56a4688-d931-48d4-9b01-eaf8f1232105-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:54.926793 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:54.926785 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt5dj\" (UniqueName: \"kubernetes.io/projected/f56a4688-d931-48d4-9b01-eaf8f1232105-kube-api-access-gt5dj\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:54.926843 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:54.926835 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b4a574-4682-4f13-8883-f55fd0f9d072-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:54.928307 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:54.928282 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1b4a574-4682-4f13-8883-f55fd0f9d072-kube-api-access-8kvk4" pod "a1b4a574-4682-4f13-8883-f55fd0f9d072" (UID: "a1b4a574-4682-4f13-8883-f55fd0f9d072"). InnerVolumeSpecName "kube-api-access-8kvk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:29:54.999977 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:54.997331 5727 generic.go:356] "Generic (PLEG): container finished" podID="f56a4688-d931-48d4-9b01-eaf8f1232105" containerID="79198533f47032475fcf1a34870cee63bacab5f174e3001200f2b89039a02cce" exitCode=0 Nov 02 23:29:54.999977 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:54.997506 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-55ccb9d5b9-p7grv" Nov 02 23:29:54.999977 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:54.999635 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-55ccb9d5b9-p7grv" event={"ID":"f56a4688-d931-48d4-9b01-eaf8f1232105","Type":"ContainerDied","Data":"79198533f47032475fcf1a34870cee63bacab5f174e3001200f2b89039a02cce"} Nov 02 23:29:54.999977 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:54.999698 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-55ccb9d5b9-p7grv" event={"ID":"f56a4688-d931-48d4-9b01-eaf8f1232105","Type":"ContainerDied","Data":"0b45fec4636b4a8d06529316113286782c311d80de513802932043f1f19c069a"} Nov 02 23:29:54.999977 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:54.999729 5727 scope.go:122] "RemoveContainer" containerID="79198533f47032475fcf1a34870cee63bacab5f174e3001200f2b89039a02cce" Nov 02 23:29:55.003342 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:55.003308 5727 generic.go:356] "Generic (PLEG): container finished" podID="a1b4a574-4682-4f13-8883-f55fd0f9d072" containerID="63f66534616e04ce6f7e4bd417fabee15636cbff0d81e48b2841e5e4abbd0650" exitCode=0 Nov 02 23:29:55.003467 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:55.003360 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-57f5cf78bc-9ckl7" Nov 02 23:29:55.004397 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:55.004364 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-57f5cf78bc-9ckl7" event={"ID":"a1b4a574-4682-4f13-8883-f55fd0f9d072","Type":"ContainerDied","Data":"63f66534616e04ce6f7e4bd417fabee15636cbff0d81e48b2841e5e4abbd0650"} Nov 02 23:29:55.004496 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:55.004425 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-57f5cf78bc-9ckl7" event={"ID":"a1b4a574-4682-4f13-8883-f55fd0f9d072","Type":"ContainerDied","Data":"acfce5f0ab90964732d7782353e0779956220cdfc0f09174d919cf13dcd0cd35"} Nov 02 23:29:55.019957 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:55.019939 5727 scope.go:122] "RemoveContainer" containerID="79198533f47032475fcf1a34870cee63bacab5f174e3001200f2b89039a02cce" Nov 02 23:29:55.020368 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:29:55.020340 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79198533f47032475fcf1a34870cee63bacab5f174e3001200f2b89039a02cce\": container with ID starting with 79198533f47032475fcf1a34870cee63bacab5f174e3001200f2b89039a02cce not found: ID does not exist" containerID="79198533f47032475fcf1a34870cee63bacab5f174e3001200f2b89039a02cce" Nov 02 23:29:55.020469 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:55.020381 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79198533f47032475fcf1a34870cee63bacab5f174e3001200f2b89039a02cce"} err="failed to get container status \"79198533f47032475fcf1a34870cee63bacab5f174e3001200f2b89039a02cce\": rpc error: code = NotFound desc = could not find container \"79198533f47032475fcf1a34870cee63bacab5f174e3001200f2b89039a02cce\": container with ID starting with 79198533f47032475fcf1a34870cee63bacab5f174e3001200f2b89039a02cce not found: ID does not exist" Nov 02 23:29:55.020469 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:55.020405 5727 scope.go:122] "RemoveContainer" containerID="63f66534616e04ce6f7e4bd417fabee15636cbff0d81e48b2841e5e4abbd0650" Nov 02 23:29:55.027942 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:55.027912 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kvk4\" (UniqueName: \"kubernetes.io/projected/a1b4a574-4682-4f13-8883-f55fd0f9d072-kube-api-access-8kvk4\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:55.032773 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:55.032741 5727 scope.go:122] "RemoveContainer" containerID="63f66534616e04ce6f7e4bd417fabee15636cbff0d81e48b2841e5e4abbd0650" Nov 02 23:29:55.033480 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:29:55.033456 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63f66534616e04ce6f7e4bd417fabee15636cbff0d81e48b2841e5e4abbd0650\": container with ID starting with 63f66534616e04ce6f7e4bd417fabee15636cbff0d81e48b2841e5e4abbd0650 not found: ID does not exist" containerID="63f66534616e04ce6f7e4bd417fabee15636cbff0d81e48b2841e5e4abbd0650" Nov 02 23:29:55.033551 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:55.033492 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63f66534616e04ce6f7e4bd417fabee15636cbff0d81e48b2841e5e4abbd0650"} err="failed to get container status \"63f66534616e04ce6f7e4bd417fabee15636cbff0d81e48b2841e5e4abbd0650\": rpc error: code = NotFound desc = could not find container \"63f66534616e04ce6f7e4bd417fabee15636cbff0d81e48b2841e5e4abbd0650\": container with ID starting with 63f66534616e04ce6f7e4bd417fabee15636cbff0d81e48b2841e5e4abbd0650 not found: ID does not exist" Nov 02 23:29:55.191779 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:55.191675 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-818fe8-55ccb9d5b9-p7grv"] Nov 02 23:29:55.196443 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:55.196415 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-818fe8-55ccb9d5b9-p7grv"] Nov 02 23:29:55.290582 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:55.290507 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-57f5cf78bc-9ckl7"] Nov 02 23:29:55.293962 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:55.293940 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-57f5cf78bc-9ckl7"] Nov 02 23:29:55.739654 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:55.739617 5727 patch_prober.go:29] interesting pod/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 02 23:29:55.739798 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:55.739670 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" podUID="c5c0fb24-5f82-4c08-bc9b-f80476f4ded9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 02 23:29:55.889027 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:55.888986 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1b4a574-4682-4f13-8883-f55fd0f9d072" path="/var/lib/kubelet/pods/a1b4a574-4682-4f13-8883-f55fd0f9d072/volumes" Nov 02 23:29:55.889563 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:55.889537 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f56a4688-d931-48d4-9b01-eaf8f1232105" path="/var/lib/kubelet/pods/f56a4688-d931-48d4-9b01-eaf8f1232105/volumes" Nov 02 23:29:56.740253 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:56.740212 5727 patch_prober.go:29] interesting pod/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 02 23:29:56.740657 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:56.740267 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" podUID="c5c0fb24-5f82-4c08-bc9b-f80476f4ded9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 02 23:29:57.740314 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:57.740276 5727 patch_prober.go:29] interesting pod/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 02 23:29:57.740707 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:57.740337 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" podUID="c5c0fb24-5f82-4c08-bc9b-f80476f4ded9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 02 23:29:57.847141 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:29:57.847094 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:29:58.739929 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:58.739893 5727 patch_prober.go:29] interesting pod/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 02 23:29:58.740158 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:58.739951 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" podUID="c5c0fb24-5f82-4c08-bc9b-f80476f4ded9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 02 23:29:59.744363 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:59.744330 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:29:59.744840 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:59.744769 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:29:59.745450 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:29:59.745429 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:30:00.197518 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:00.197484 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g"] Nov 02 23:30:00.198488 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:00.198464 5727 cpu_manager.go:414] "containerMap: removing container" podUID="f56a4688-d931-48d4-9b01-eaf8f1232105" containerName="payload-processing" Nov 02 23:30:00.198488 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:00.198489 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f56a4688-d931-48d4-9b01-eaf8f1232105" containerName="payload-processing" Nov 02 23:30:00.198648 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:00.198516 5727 cpu_manager.go:414] "containerMap: removing container" podUID="a1b4a574-4682-4f13-8883-f55fd0f9d072" containerName="payload-pre-processing" Nov 02 23:30:00.198648 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:00.198525 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1b4a574-4682-4f13-8883-f55fd0f9d072" containerName="payload-pre-processing" Nov 02 23:30:00.198717 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:00.198687 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="f56a4688-d931-48d4-9b01-eaf8f1232105" containerName="payload-processing" Nov 02 23:30:00.198717 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:00.198703 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1b4a574-4682-4f13-8883-f55fd0f9d072" containerName="payload-pre-processing" Nov 02 23:30:00.202746 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:00.202728 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth"] Nov 02 23:30:00.202869 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:00.202853 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g" Nov 02 23:30:00.206778 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:00.206756 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth" Nov 02 23:30:00.279160 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:00.278777 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g"] Nov 02 23:30:00.283789 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:00.283764 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth"] Nov 02 23:30:00.325084 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:00.325054 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a463aa29-3e49-4bde-a4d2-868c217831c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth\" (UID: \"a463aa29-3e49-4bde-a4d2-868c217831c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth" Nov 02 23:30:00.325257 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:00.325175 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6md9\" (UniqueName: \"kubernetes.io/projected/30477205-05db-46c2-b474-efe14b4ecdfd-kube-api-access-g6md9\") pod \"maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g\" (UID: \"30477205-05db-46c2-b474-efe14b4ecdfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g" Nov 02 23:30:00.325257 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:00.325210 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq4jx\" (UniqueName: \"kubernetes.io/projected/a463aa29-3e49-4bde-a4d2-868c217831c6-kube-api-access-tq4jx\") pod \"maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth\" (UID: \"a463aa29-3e49-4bde-a4d2-868c217831c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth" Nov 02 23:30:00.325339 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:00.325273 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30477205-05db-46c2-b474-efe14b4ecdfd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g\" (UID: \"30477205-05db-46c2-b474-efe14b4ecdfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g" Nov 02 23:30:00.330572 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:00.330551 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:30:00.426336 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:00.426294 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a463aa29-3e49-4bde-a4d2-868c217831c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth\" (UID: \"a463aa29-3e49-4bde-a4d2-868c217831c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth" Nov 02 23:30:00.426524 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:00.426427 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6md9\" (UniqueName: \"kubernetes.io/projected/30477205-05db-46c2-b474-efe14b4ecdfd-kube-api-access-g6md9\") pod \"maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g\" (UID: \"30477205-05db-46c2-b474-efe14b4ecdfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g" Nov 02 23:30:00.426524 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:00.426476 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq4jx\" (UniqueName: \"kubernetes.io/projected/a463aa29-3e49-4bde-a4d2-868c217831c6-kube-api-access-tq4jx\") pod \"maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth\" (UID: \"a463aa29-3e49-4bde-a4d2-868c217831c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth" Nov 02 23:30:00.426661 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:00.426533 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30477205-05db-46c2-b474-efe14b4ecdfd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g\" (UID: \"30477205-05db-46c2-b474-efe14b4ecdfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g" Nov 02 23:30:00.427095 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:00.427066 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a463aa29-3e49-4bde-a4d2-868c217831c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth\" (UID: \"a463aa29-3e49-4bde-a4d2-868c217831c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth" Nov 02 23:30:00.427318 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:00.427283 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30477205-05db-46c2-b474-efe14b4ecdfd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g\" (UID: \"30477205-05db-46c2-b474-efe14b4ecdfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g" Nov 02 23:30:00.487261 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:00.487178 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6md9\" (UniqueName: \"kubernetes.io/projected/30477205-05db-46c2-b474-efe14b4ecdfd-kube-api-access-g6md9\") pod \"maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g\" (UID: \"30477205-05db-46c2-b474-efe14b4ecdfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g" Nov 02 23:30:00.487447 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:00.487426 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq4jx\" (UniqueName: \"kubernetes.io/projected/a463aa29-3e49-4bde-a4d2-868c217831c6-kube-api-access-tq4jx\") pod \"maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth\" (UID: \"a463aa29-3e49-4bde-a4d2-868c217831c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth" Nov 02 23:30:00.519859 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:00.519835 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g" Nov 02 23:30:00.526160 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:00.526116 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth" Nov 02 23:30:00.901080 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:00.901051 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g"] Nov 02 23:30:00.901454 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:30:00.901258 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30477205_05db_46c2_b474_efe14b4ecdfd.slice/crio-94fe5cb7205fe8fd44543bac7060016c50b8180981ba1952c1a18a7432b8b088 WatchSource:0}: Error finding container 94fe5cb7205fe8fd44543bac7060016c50b8180981ba1952c1a18a7432b8b088: Status 404 returned error can't find the container with id 94fe5cb7205fe8fd44543bac7060016c50b8180981ba1952c1a18a7432b8b088 Nov 02 23:30:00.951796 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:00.951770 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth"] Nov 02 23:30:00.952522 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:30:00.952499 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda463aa29_3e49_4bde_a4d2_868c217831c6.slice/crio-6c105d074bae56b94ac767cfafb8115953bea3886bab0c2727d9aa7fddb32b32 WatchSource:0}: Error finding container 6c105d074bae56b94ac767cfafb8115953bea3886bab0c2727d9aa7fddb32b32: Status 404 returned error can't find the container with id 6c105d074bae56b94ac767cfafb8115953bea3886bab0c2727d9aa7fddb32b32 Nov 02 23:30:01.056680 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:01.056652 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g" event={"ID":"30477205-05db-46c2-b474-efe14b4ecdfd","Type":"ContainerStarted","Data":"94fe5cb7205fe8fd44543bac7060016c50b8180981ba1952c1a18a7432b8b088"} Nov 02 23:30:01.058262 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:01.058243 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth" event={"ID":"a463aa29-3e49-4bde-a4d2-868c217831c6","Type":"ContainerStarted","Data":"6c105d074bae56b94ac767cfafb8115953bea3886bab0c2727d9aa7fddb32b32"} Nov 02 23:30:02.068018 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:02.067975 5727 generic.go:356] "Generic (PLEG): container finished" podID="a463aa29-3e49-4bde-a4d2-868c217831c6" containerID="814da2de06d42115f53b08c0a917554dfddcffbe46039c41dcee51909087124b" exitCode=60 Nov 02 23:30:02.068576 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:02.068388 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth" event={"ID":"a463aa29-3e49-4bde-a4d2-868c217831c6","Type":"ContainerDied","Data":"814da2de06d42115f53b08c0a917554dfddcffbe46039c41dcee51909087124b"} Nov 02 23:30:02.068576 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:02.068466 5727 scope.go:122] "RemoveContainer" containerID="814da2de06d42115f53b08c0a917554dfddcffbe46039c41dcee51909087124b" Nov 02 23:30:02.071601 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:02.071577 5727 generic.go:356] "Generic (PLEG): container finished" podID="30477205-05db-46c2-b474-efe14b4ecdfd" containerID="b205e3b6c1d57e10fb0d15be57fa7f5b6a8e3c7448d39971294dff420918f40d" exitCode=60 Nov 02 23:30:02.071730 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:02.071721 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g" event={"ID":"30477205-05db-46c2-b474-efe14b4ecdfd","Type":"ContainerDied","Data":"b205e3b6c1d57e10fb0d15be57fa7f5b6a8e3c7448d39971294dff420918f40d"} Nov 02 23:30:02.072023 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:02.072006 5727 scope.go:122] "RemoveContainer" containerID="b205e3b6c1d57e10fb0d15be57fa7f5b6a8e3c7448d39971294dff420918f40d" Nov 02 23:30:03.082498 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:03.082467 5727 generic.go:356] "Generic (PLEG): container finished" podID="a463aa29-3e49-4bde-a4d2-868c217831c6" containerID="245efc7893ed608df61a5c5b5d65dc8773b8e8c174baf96fb0b5f28180b38c4b" exitCode=60 Nov 02 23:30:03.082927 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:03.082511 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth" event={"ID":"a463aa29-3e49-4bde-a4d2-868c217831c6","Type":"ContainerDied","Data":"245efc7893ed608df61a5c5b5d65dc8773b8e8c174baf96fb0b5f28180b38c4b"} Nov 02 23:30:03.082927 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:03.082574 5727 scope.go:122] "RemoveContainer" containerID="814da2de06d42115f53b08c0a917554dfddcffbe46039c41dcee51909087124b" Nov 02 23:30:03.082927 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:03.082864 5727 scope.go:122] "RemoveContainer" containerID="245efc7893ed608df61a5c5b5d65dc8773b8e8c174baf96fb0b5f28180b38c4b" Nov 02 23:30:03.083267 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:30:03.083240 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth_odh-ai-gateway-infra(a463aa29-3e49-4bde-a4d2-868c217831c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth" podUID="a463aa29-3e49-4bde-a4d2-868c217831c6" Nov 02 23:30:03.084945 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:03.084923 5727 generic.go:356] "Generic (PLEG): container finished" podID="30477205-05db-46c2-b474-efe14b4ecdfd" containerID="63ec89142df60ab60aa4556aa14e4ef6d59b059c12a195522a0d718fb48a8481" exitCode=60 Nov 02 23:30:03.085032 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:03.085005 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g" event={"ID":"30477205-05db-46c2-b474-efe14b4ecdfd","Type":"ContainerDied","Data":"63ec89142df60ab60aa4556aa14e4ef6d59b059c12a195522a0d718fb48a8481"} Nov 02 23:30:03.085256 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:03.085242 5727 scope.go:122] "RemoveContainer" containerID="63ec89142df60ab60aa4556aa14e4ef6d59b059c12a195522a0d718fb48a8481" Nov 02 23:30:03.085508 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:30:03.085484 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g_odh-ai-gateway-infra(30477205-05db-46c2-b474-efe14b4ecdfd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g" podUID="30477205-05db-46c2-b474-efe14b4ecdfd" Nov 02 23:30:03.108313 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:03.108291 5727 scope.go:122] "RemoveContainer" containerID="b205e3b6c1d57e10fb0d15be57fa7f5b6a8e3c7448d39971294dff420918f40d" Nov 02 23:30:04.099911 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:04.099882 5727 scope.go:122] "RemoveContainer" containerID="245efc7893ed608df61a5c5b5d65dc8773b8e8c174baf96fb0b5f28180b38c4b" Nov 02 23:30:04.100445 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:30:04.100218 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth_odh-ai-gateway-infra(a463aa29-3e49-4bde-a4d2-868c217831c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth" podUID="a463aa29-3e49-4bde-a4d2-868c217831c6" Nov 02 23:30:04.101945 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:04.101905 5727 scope.go:122] "RemoveContainer" containerID="63ec89142df60ab60aa4556aa14e4ef6d59b059c12a195522a0d718fb48a8481" Nov 02 23:30:04.102393 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:30:04.102370 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g_odh-ai-gateway-infra(30477205-05db-46c2-b474-efe14b4ecdfd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g" podUID="30477205-05db-46c2-b474-efe14b4ecdfd" Nov 02 23:30:12.158074 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:12.158031 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c84dbcbd-w72p6"] Nov 02 23:30:12.164660 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:12.164637 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84dbcbd-w72p6" Nov 02 23:30:12.218545 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:12.218514 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c84dbcbd-w72p6"] Nov 02 23:30:12.263568 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:12.263531 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:30:12.274346 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:12.274315 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bblg\" (UniqueName: \"kubernetes.io/projected/843c2835-6862-44c8-ac4e-2d5cd3806fa8-kube-api-access-9bblg\") pod \"maas-api-revoke-sub-c84dbcbd-w72p6\" (UID: \"843c2835-6862-44c8-ac4e-2d5cd3806fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84dbcbd-w72p6" Nov 02 23:30:12.274489 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:12.274378 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843c2835-6862-44c8-ac4e-2d5cd3806fa8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c84dbcbd-w72p6\" (UID: \"843c2835-6862-44c8-ac4e-2d5cd3806fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84dbcbd-w72p6" Nov 02 23:30:12.274576 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:12.274519 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843c2835-6862-44c8-ac4e-2d5cd3806fa8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c84dbcbd-w72p6\" (UID: \"843c2835-6862-44c8-ac4e-2d5cd3806fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84dbcbd-w72p6" Nov 02 23:30:12.376166 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:12.376109 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bblg\" (UniqueName: \"kubernetes.io/projected/843c2835-6862-44c8-ac4e-2d5cd3806fa8-kube-api-access-9bblg\") pod \"maas-api-revoke-sub-c84dbcbd-w72p6\" (UID: \"843c2835-6862-44c8-ac4e-2d5cd3806fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84dbcbd-w72p6" Nov 02 23:30:12.376344 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:12.376199 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843c2835-6862-44c8-ac4e-2d5cd3806fa8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c84dbcbd-w72p6\" (UID: \"843c2835-6862-44c8-ac4e-2d5cd3806fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84dbcbd-w72p6" Nov 02 23:30:12.376344 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:12.376286 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843c2835-6862-44c8-ac4e-2d5cd3806fa8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c84dbcbd-w72p6\" (UID: \"843c2835-6862-44c8-ac4e-2d5cd3806fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84dbcbd-w72p6" Nov 02 23:30:12.376879 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:12.376858 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843c2835-6862-44c8-ac4e-2d5cd3806fa8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c84dbcbd-w72p6\" (UID: \"843c2835-6862-44c8-ac4e-2d5cd3806fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84dbcbd-w72p6" Nov 02 23:30:12.378860 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:12.378837 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843c2835-6862-44c8-ac4e-2d5cd3806fa8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c84dbcbd-w72p6\" (UID: \"843c2835-6862-44c8-ac4e-2d5cd3806fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84dbcbd-w72p6" Nov 02 23:30:12.436963 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:12.436896 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bblg\" (UniqueName: \"kubernetes.io/projected/843c2835-6862-44c8-ac4e-2d5cd3806fa8-kube-api-access-9bblg\") pod \"maas-api-revoke-sub-c84dbcbd-w72p6\" (UID: \"843c2835-6862-44c8-ac4e-2d5cd3806fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84dbcbd-w72p6" Nov 02 23:30:12.482218 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:12.482170 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84dbcbd-w72p6" Nov 02 23:30:12.879229 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:12.879188 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c84dbcbd-w72p6"] Nov 02 23:30:12.880539 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:30:12.880509 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod843c2835_6862_44c8_ac4e_2d5cd3806fa8.slice/crio-b2cbf60be3e606904fe43d2d01a15d9aa12c55af90fae602b0df192112bed1df WatchSource:0}: Error finding container b2cbf60be3e606904fe43d2d01a15d9aa12c55af90fae602b0df192112bed1df: Status 404 returned error can't find the container with id b2cbf60be3e606904fe43d2d01a15d9aa12c55af90fae602b0df192112bed1df Nov 02 23:30:13.182833 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:13.182804 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84dbcbd-w72p6" event={"ID":"843c2835-6862-44c8-ac4e-2d5cd3806fa8","Type":"ContainerStarted","Data":"62215ffc337101c9d54db54c8b3b900c0686d18c7029fa64bd772b6154c44720"} Nov 02 23:30:13.183247 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:13.182856 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84dbcbd-w72p6" event={"ID":"843c2835-6862-44c8-ac4e-2d5cd3806fa8","Type":"ContainerStarted","Data":"b2cbf60be3e606904fe43d2d01a15d9aa12c55af90fae602b0df192112bed1df"} Nov 02 23:30:13.299787 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:13.299726 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84dbcbd-w72p6" podStartSLOduration=1.2997069780000001 podStartE2EDuration="1.299706978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:13.297117893 +0000 UTC m=+2499.535615556" watchObservedRunningTime="2026-11-02 23:30:13.299706978 +0000 UTC m=+2499.538204640" Nov 02 23:30:14.192550 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:14.192511 5727 generic.go:356] "Generic (PLEG): container finished" podID="843c2835-6862-44c8-ac4e-2d5cd3806fa8" containerID="62215ffc337101c9d54db54c8b3b900c0686d18c7029fa64bd772b6154c44720" exitCode=0 Nov 02 23:30:14.192945 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:14.192597 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84dbcbd-w72p6" event={"ID":"843c2835-6862-44c8-ac4e-2d5cd3806fa8","Type":"ContainerDied","Data":"62215ffc337101c9d54db54c8b3b900c0686d18c7029fa64bd772b6154c44720"} Nov 02 23:30:14.875627 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:14.875581 5727 scope.go:122] "RemoveContainer" containerID="245efc7893ed608df61a5c5b5d65dc8773b8e8c174baf96fb0b5f28180b38c4b" Nov 02 23:30:15.366204 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:15.366176 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84dbcbd-w72p6" Nov 02 23:30:15.427073 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:15.427045 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/843c2835-6862-44c8-ac4e-2d5cd3806fa8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843c2835-6862-44c8-ac4e-2d5cd3806fa8-maas-api-xks-ca\") pod \"843c2835-6862-44c8-ac4e-2d5cd3806fa8\" (UID: \"843c2835-6862-44c8-ac4e-2d5cd3806fa8\") " Nov 02 23:30:15.427254 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:15.427161 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/843c2835-6862-44c8-ac4e-2d5cd3806fa8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843c2835-6862-44c8-ac4e-2d5cd3806fa8-maas-api-openshift-ca\") pod \"843c2835-6862-44c8-ac4e-2d5cd3806fa8\" (UID: \"843c2835-6862-44c8-ac4e-2d5cd3806fa8\") " Nov 02 23:30:15.427254 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:15.427201 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/843c2835-6862-44c8-ac4e-2d5cd3806fa8-kube-api-access-9bblg\" (UniqueName: \"kubernetes.io/projected/843c2835-6862-44c8-ac4e-2d5cd3806fa8-kube-api-access-9bblg\") pod \"843c2835-6862-44c8-ac4e-2d5cd3806fa8\" (UID: \"843c2835-6862-44c8-ac4e-2d5cd3806fa8\") " Nov 02 23:30:15.427554 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:15.427530 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/843c2835-6862-44c8-ac4e-2d5cd3806fa8-maas-api-openshift-ca" pod "843c2835-6862-44c8-ac4e-2d5cd3806fa8" (UID: "843c2835-6862-44c8-ac4e-2d5cd3806fa8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:15.427763 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:15.427745 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843c2835-6862-44c8-ac4e-2d5cd3806fa8-maas-api-openshift-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:15.429422 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:15.429398 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/843c2835-6862-44c8-ac4e-2d5cd3806fa8-maas-api-xks-ca" pod "843c2835-6862-44c8-ac4e-2d5cd3806fa8" (UID: "843c2835-6862-44c8-ac4e-2d5cd3806fa8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:30:15.429550 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:15.429531 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/843c2835-6862-44c8-ac4e-2d5cd3806fa8-kube-api-access-9bblg" pod "843c2835-6862-44c8-ac4e-2d5cd3806fa8" (UID: "843c2835-6862-44c8-ac4e-2d5cd3806fa8"). InnerVolumeSpecName "kube-api-access-9bblg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:15.528531 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:15.528453 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843c2835-6862-44c8-ac4e-2d5cd3806fa8-maas-api-xks-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:15.528531 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:15.528483 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bblg\" (UniqueName: \"kubernetes.io/projected/843c2835-6862-44c8-ac4e-2d5cd3806fa8-kube-api-access-9bblg\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:16.209597 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:16.209513 5727 generic.go:356] "Generic (PLEG): container finished" podID="a463aa29-3e49-4bde-a4d2-868c217831c6" containerID="73d0f56d66de209e493213ffc192bfc755f2fb2b8e5fdd1fb4fe059ca3499aef" exitCode=60 Nov 02 23:30:16.209791 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:16.209613 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth" event={"ID":"a463aa29-3e49-4bde-a4d2-868c217831c6","Type":"ContainerDied","Data":"73d0f56d66de209e493213ffc192bfc755f2fb2b8e5fdd1fb4fe059ca3499aef"} Nov 02 23:30:16.209791 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:16.209667 5727 scope.go:122] "RemoveContainer" containerID="245efc7893ed608df61a5c5b5d65dc8773b8e8c174baf96fb0b5f28180b38c4b" Nov 02 23:30:16.210187 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:16.210164 5727 scope.go:122] "RemoveContainer" containerID="73d0f56d66de209e493213ffc192bfc755f2fb2b8e5fdd1fb4fe059ca3499aef" Nov 02 23:30:16.210533 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:30:16.210504 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth_odh-ai-gateway-infra(a463aa29-3e49-4bde-a4d2-868c217831c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth" podUID="a463aa29-3e49-4bde-a4d2-868c217831c6" Nov 02 23:30:16.211962 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:16.211941 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84dbcbd-w72p6" Nov 02 23:30:16.212048 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:16.211977 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84dbcbd-w72p6" event={"ID":"843c2835-6862-44c8-ac4e-2d5cd3806fa8","Type":"ContainerDied","Data":"b2cbf60be3e606904fe43d2d01a15d9aa12c55af90fae602b0df192112bed1df"} Nov 02 23:30:16.212048 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:16.212020 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2cbf60be3e606904fe43d2d01a15d9aa12c55af90fae602b0df192112bed1df" Nov 02 23:30:17.338276 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:17.338240 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth"] Nov 02 23:30:17.521681 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:17.521659 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth" Nov 02 23:30:17.658058 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:17.657965 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a463aa29-3e49-4bde-a4d2-868c217831c6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a463aa29-3e49-4bde-a4d2-868c217831c6-maas-api-service-ca\") pod \"a463aa29-3e49-4bde-a4d2-868c217831c6\" (UID: \"a463aa29-3e49-4bde-a4d2-868c217831c6\") " Nov 02 23:30:17.658261 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:17.658196 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a463aa29-3e49-4bde-a4d2-868c217831c6-kube-api-access-tq4jx\" (UniqueName: \"kubernetes.io/projected/a463aa29-3e49-4bde-a4d2-868c217831c6-kube-api-access-tq4jx\") pod \"a463aa29-3e49-4bde-a4d2-868c217831c6\" (UID: \"a463aa29-3e49-4bde-a4d2-868c217831c6\") " Nov 02 23:30:17.658459 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:17.658435 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a463aa29-3e49-4bde-a4d2-868c217831c6-maas-api-service-ca" pod "a463aa29-3e49-4bde-a4d2-868c217831c6" (UID: "a463aa29-3e49-4bde-a4d2-868c217831c6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:17.658677 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:17.658662 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a463aa29-3e49-4bde-a4d2-868c217831c6-maas-api-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:17.660602 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:17.660575 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a463aa29-3e49-4bde-a4d2-868c217831c6-kube-api-access-tq4jx" pod "a463aa29-3e49-4bde-a4d2-868c217831c6" (UID: "a463aa29-3e49-4bde-a4d2-868c217831c6"). InnerVolumeSpecName "kube-api-access-tq4jx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:17.760618 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:17.760573 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq4jx\" (UniqueName: \"kubernetes.io/projected/a463aa29-3e49-4bde-a4d2-868c217831c6-kube-api-access-tq4jx\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:17.885070 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:17.885038 5727 scope.go:122] "RemoveContainer" containerID="63ec89142df60ab60aa4556aa14e4ef6d59b059c12a195522a0d718fb48a8481" Nov 02 23:30:18.229720 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:18.229640 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth" Nov 02 23:30:18.229720 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:18.229677 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth" event={"ID":"a463aa29-3e49-4bde-a4d2-868c217831c6","Type":"ContainerDied","Data":"6c105d074bae56b94ac767cfafb8115953bea3886bab0c2727d9aa7fddb32b32"} Nov 02 23:30:18.229933 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:18.229758 5727 scope.go:122] "RemoveContainer" containerID="73d0f56d66de209e493213ffc192bfc755f2fb2b8e5fdd1fb4fe059ca3499aef" Nov 02 23:30:18.232190 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:18.232159 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g" event={"ID":"30477205-05db-46c2-b474-efe14b4ecdfd","Type":"ContainerStarted","Data":"9d7f3641dff892baf1b01b024997effe8beade44345e894b6ec6244127e42c54"} Nov 02 23:30:18.398272 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:18.398217 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g" podStartSLOduration=18.398202806 podStartE2EDuration="18.398202806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:18.396049906 +0000 UTC m=+2504.634547607" watchObservedRunningTime="2026-11-02 23:30:18.398202806 +0000 UTC m=+2504.636700467" Nov 02 23:30:18.510781 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:18.510698 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth"] Nov 02 23:30:18.518588 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:18.518557 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-adfeab-29894370-jsxth"] Nov 02 23:30:19.242624 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:19.242593 5727 generic.go:356] "Generic (PLEG): container finished" podID="30477205-05db-46c2-b474-efe14b4ecdfd" containerID="9d7f3641dff892baf1b01b024997effe8beade44345e894b6ec6244127e42c54" exitCode=60 Nov 02 23:30:19.242791 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:19.242669 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g" event={"ID":"30477205-05db-46c2-b474-efe14b4ecdfd","Type":"ContainerDied","Data":"9d7f3641dff892baf1b01b024997effe8beade44345e894b6ec6244127e42c54"} Nov 02 23:30:19.242791 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:19.242738 5727 scope.go:122] "RemoveContainer" containerID="63ec89142df60ab60aa4556aa14e4ef6d59b059c12a195522a0d718fb48a8481" Nov 02 23:30:19.243144 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:19.243104 5727 scope.go:122] "RemoveContainer" containerID="9d7f3641dff892baf1b01b024997effe8beade44345e894b6ec6244127e42c54" Nov 02 23:30:19.243452 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:30:19.243432 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g_odh-ai-gateway-infra(30477205-05db-46c2-b474-efe14b4ecdfd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g" podUID="30477205-05db-46c2-b474-efe14b4ecdfd" Nov 02 23:30:19.406941 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:19.406907 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g"] Nov 02 23:30:19.478608 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:19.478579 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4"] Nov 02 23:30:19.478868 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:19.478816 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4" podUID="b57e0c31-0e6f-456a-9855-cb5d9c083c88" containerName="maas-api" containerID="cri-o://3629ebb2c14fa7dba37dd92396eb48aa5888cbc4b34325b8360392d6b4ada486" gracePeriod=30 Nov 02 23:30:19.772254 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:19.772191 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4" Nov 02 23:30:19.885151 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:19.883588 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b57e0c31-0e6f-456a-9855-cb5d9c083c88-kube-api-access-mxd27\" (UniqueName: \"kubernetes.io/projected/b57e0c31-0e6f-456a-9855-cb5d9c083c88-kube-api-access-mxd27\") pod \"b57e0c31-0e6f-456a-9855-cb5d9c083c88\" (UID: \"b57e0c31-0e6f-456a-9855-cb5d9c083c88\") " Nov 02 23:30:19.885151 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:19.883635 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b57e0c31-0e6f-456a-9855-cb5d9c083c88-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b57e0c31-0e6f-456a-9855-cb5d9c083c88-maas-api-tls\") pod \"b57e0c31-0e6f-456a-9855-cb5d9c083c88\" (UID: \"b57e0c31-0e6f-456a-9855-cb5d9c083c88\") " Nov 02 23:30:19.885151 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:19.883681 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b57e0c31-0e6f-456a-9855-cb5d9c083c88-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b57e0c31-0e6f-456a-9855-cb5d9c083c88-openshift-service-ca\") pod \"b57e0c31-0e6f-456a-9855-cb5d9c083c88\" (UID: \"b57e0c31-0e6f-456a-9855-cb5d9c083c88\") " Nov 02 23:30:19.885151 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:19.883747 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b57e0c31-0e6f-456a-9855-cb5d9c083c88-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b57e0c31-0e6f-456a-9855-cb5d9c083c88-metrics-certs\") pod \"b57e0c31-0e6f-456a-9855-cb5d9c083c88\" (UID: \"b57e0c31-0e6f-456a-9855-cb5d9c083c88\") " Nov 02 23:30:19.885526 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:19.885171 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b57e0c31-0e6f-456a-9855-cb5d9c083c88-openshift-service-ca" pod "b57e0c31-0e6f-456a-9855-cb5d9c083c88" (UID: "b57e0c31-0e6f-456a-9855-cb5d9c083c88"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:19.905794 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:19.905271 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a463aa29-3e49-4bde-a4d2-868c217831c6" path="/var/lib/kubelet/pods/a463aa29-3e49-4bde-a4d2-868c217831c6/volumes" Nov 02 23:30:19.906515 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:19.906475 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b57e0c31-0e6f-456a-9855-cb5d9c083c88-maas-api-tls" pod "b57e0c31-0e6f-456a-9855-cb5d9c083c88" (UID: "b57e0c31-0e6f-456a-9855-cb5d9c083c88"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:30:19.909395 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:19.909368 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b57e0c31-0e6f-456a-9855-cb5d9c083c88-metrics-certs" pod "b57e0c31-0e6f-456a-9855-cb5d9c083c88" (UID: "b57e0c31-0e6f-456a-9855-cb5d9c083c88"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:30:19.909395 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:19.909364 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b57e0c31-0e6f-456a-9855-cb5d9c083c88-kube-api-access-mxd27" pod "b57e0c31-0e6f-456a-9855-cb5d9c083c88" (UID: "b57e0c31-0e6f-456a-9855-cb5d9c083c88"). InnerVolumeSpecName "kube-api-access-mxd27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:19.985971 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:19.985930 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxd27\" (UniqueName: \"kubernetes.io/projected/b57e0c31-0e6f-456a-9855-cb5d9c083c88-kube-api-access-mxd27\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:19.986186 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:19.986041 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b57e0c31-0e6f-456a-9855-cb5d9c083c88-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:19.986186 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:19.986063 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b57e0c31-0e6f-456a-9855-cb5d9c083c88-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:19.986186 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:19.986079 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b57e0c31-0e6f-456a-9855-cb5d9c083c88-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:20.253850 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:20.253816 5727 generic.go:356] "Generic (PLEG): container finished" podID="b57e0c31-0e6f-456a-9855-cb5d9c083c88" containerID="3629ebb2c14fa7dba37dd92396eb48aa5888cbc4b34325b8360392d6b4ada486" exitCode=0 Nov 02 23:30:20.253969 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:20.253863 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4" Nov 02 23:30:20.253969 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:20.253903 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4" event={"ID":"b57e0c31-0e6f-456a-9855-cb5d9c083c88","Type":"ContainerDied","Data":"3629ebb2c14fa7dba37dd92396eb48aa5888cbc4b34325b8360392d6b4ada486"} Nov 02 23:30:20.254086 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:20.253978 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4" event={"ID":"b57e0c31-0e6f-456a-9855-cb5d9c083c88","Type":"ContainerDied","Data":"3955e2a1a8ad5654b23eebb92c3c0f11ef84564adba8fc6c1c126dab53f8a152"} Nov 02 23:30:20.254086 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:20.254005 5727 scope.go:122] "RemoveContainer" containerID="3629ebb2c14fa7dba37dd92396eb48aa5888cbc4b34325b8360392d6b4ada486" Nov 02 23:30:20.268148 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:20.268103 5727 scope.go:122] "RemoveContainer" containerID="3629ebb2c14fa7dba37dd92396eb48aa5888cbc4b34325b8360392d6b4ada486" Nov 02 23:30:20.268521 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:30:20.268491 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3629ebb2c14fa7dba37dd92396eb48aa5888cbc4b34325b8360392d6b4ada486\": container with ID starting with 3629ebb2c14fa7dba37dd92396eb48aa5888cbc4b34325b8360392d6b4ada486 not found: ID does not exist" containerID="3629ebb2c14fa7dba37dd92396eb48aa5888cbc4b34325b8360392d6b4ada486" Nov 02 23:30:20.268626 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:20.268533 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3629ebb2c14fa7dba37dd92396eb48aa5888cbc4b34325b8360392d6b4ada486"} err="failed to get container status \"3629ebb2c14fa7dba37dd92396eb48aa5888cbc4b34325b8360392d6b4ada486\": rpc error: code = NotFound desc = could not find container \"3629ebb2c14fa7dba37dd92396eb48aa5888cbc4b34325b8360392d6b4ada486\": container with ID starting with 3629ebb2c14fa7dba37dd92396eb48aa5888cbc4b34325b8360392d6b4ada486 not found: ID does not exist" Nov 02 23:30:20.425280 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:20.425256 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g" Nov 02 23:30:20.500154 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:20.494198 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4"] Nov 02 23:30:20.500154 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:20.494687 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30477205-05db-46c2-b474-efe14b4ecdfd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30477205-05db-46c2-b474-efe14b4ecdfd-maas-api-service-ca\") pod \"30477205-05db-46c2-b474-efe14b4ecdfd\" (UID: \"30477205-05db-46c2-b474-efe14b4ecdfd\") " Nov 02 23:30:20.500154 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:20.494956 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30477205-05db-46c2-b474-efe14b4ecdfd-kube-api-access-g6md9\" (UniqueName: \"kubernetes.io/projected/30477205-05db-46c2-b474-efe14b4ecdfd-kube-api-access-g6md9\") pod \"30477205-05db-46c2-b474-efe14b4ecdfd\" (UID: \"30477205-05db-46c2-b474-efe14b4ecdfd\") " Nov 02 23:30:20.500154 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:20.498744 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30477205-05db-46c2-b474-efe14b4ecdfd-maas-api-service-ca" pod "30477205-05db-46c2-b474-efe14b4ecdfd" (UID: "30477205-05db-46c2-b474-efe14b4ecdfd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:20.507790 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:20.507699 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-85efc6-c59979788-wjjq4"] Nov 02 23:30:20.509171 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:20.509092 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30477205-05db-46c2-b474-efe14b4ecdfd-kube-api-access-g6md9" pod "30477205-05db-46c2-b474-efe14b4ecdfd" (UID: "30477205-05db-46c2-b474-efe14b4ecdfd"). InnerVolumeSpecName "kube-api-access-g6md9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:20.600027 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:20.599990 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6md9\" (UniqueName: \"kubernetes.io/projected/30477205-05db-46c2-b474-efe14b4ecdfd-kube-api-access-g6md9\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:20.600027 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:20.600027 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30477205-05db-46c2-b474-efe14b4ecdfd-maas-api-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:21.265047 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:21.265016 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g" Nov 02 23:30:21.265047 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:21.265024 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g" event={"ID":"30477205-05db-46c2-b474-efe14b4ecdfd","Type":"ContainerDied","Data":"94fe5cb7205fe8fd44543bac7060016c50b8180981ba1952c1a18a7432b8b088"} Nov 02 23:30:21.265344 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:21.265108 5727 scope.go:122] "RemoveContainer" containerID="9d7f3641dff892baf1b01b024997effe8beade44345e894b6ec6244127e42c54" Nov 02 23:30:21.430092 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:21.430056 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g"] Nov 02 23:30:21.440401 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:21.440369 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894370-gdv7g"] Nov 02 23:30:21.899816 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:21.899780 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30477205-05db-46c2-b474-efe14b4ecdfd" path="/var/lib/kubelet/pods/30477205-05db-46c2-b474-efe14b4ecdfd/volumes" Nov 02 23:30:21.900299 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:21.900283 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b57e0c31-0e6f-456a-9855-cb5d9c083c88" path="/var/lib/kubelet/pods/b57e0c31-0e6f-456a-9855-cb5d9c083c88/volumes" Nov 02 23:30:22.785741 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:22.785711 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-279zt_f36665e4-38c7-4d93-b047-306185b1a17f/payload-processing/0.log" Nov 02 23:30:23.025474 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:23.025448 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-279zt_f36665e4-38c7-4d93-b047-306185b1a17f/payload-processing/0.log" Nov 02 23:30:27.846300 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:30:27.846259 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:30:34.690670 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:34.690632 5727 scope.go:122] "RemoveContainer" containerID="bf5ddc9a741fa2b1010aa119aa19c828bd94bdb55cc2006a5fb6cb5d2ac2657b" Nov 02 23:30:56.251827 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.251747 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz"] Nov 02 23:30:56.252388 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.252374 5727 cpu_manager.go:414] "containerMap: removing container" podUID="a463aa29-3e49-4bde-a4d2-868c217831c6" containerName="cleanup" Nov 02 23:30:56.252441 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.252390 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a463aa29-3e49-4bde-a4d2-868c217831c6" containerName="cleanup" Nov 02 23:30:56.252441 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.252409 5727 cpu_manager.go:414] "containerMap: removing container" podUID="30477205-05db-46c2-b474-efe14b4ecdfd" containerName="cleanup" Nov 02 23:30:56.252441 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.252413 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30477205-05db-46c2-b474-efe14b4ecdfd" containerName="cleanup" Nov 02 23:30:56.252441 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.252433 5727 cpu_manager.go:414] "containerMap: removing container" podUID="843c2835-6862-44c8-ac4e-2d5cd3806fa8" containerName="revoke-keys" Nov 02 23:30:56.252441 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.252440 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="843c2835-6862-44c8-ac4e-2d5cd3806fa8" containerName="revoke-keys" Nov 02 23:30:56.252596 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.252449 5727 cpu_manager.go:414] "containerMap: removing container" podUID="a463aa29-3e49-4bde-a4d2-868c217831c6" containerName="cleanup" Nov 02 23:30:56.252596 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.252454 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a463aa29-3e49-4bde-a4d2-868c217831c6" containerName="cleanup" Nov 02 23:30:56.252596 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.252461 5727 cpu_manager.go:414] "containerMap: removing container" podUID="30477205-05db-46c2-b474-efe14b4ecdfd" containerName="cleanup" Nov 02 23:30:56.252596 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.252466 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30477205-05db-46c2-b474-efe14b4ecdfd" containerName="cleanup" Nov 02 23:30:56.252596 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.252475 5727 cpu_manager.go:414] "containerMap: removing container" podUID="b57e0c31-0e6f-456a-9855-cb5d9c083c88" containerName="maas-api" Nov 02 23:30:56.252596 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.252479 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b57e0c31-0e6f-456a-9855-cb5d9c083c88" containerName="maas-api" Nov 02 23:30:56.252596 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.252567 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="843c2835-6862-44c8-ac4e-2d5cd3806fa8" containerName="revoke-keys" Nov 02 23:30:56.252596 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.252576 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="a463aa29-3e49-4bde-a4d2-868c217831c6" containerName="cleanup" Nov 02 23:30:56.252596 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.252581 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="30477205-05db-46c2-b474-efe14b4ecdfd" containerName="cleanup" Nov 02 23:30:56.252596 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.252588 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="b57e0c31-0e6f-456a-9855-cb5d9c083c88" containerName="maas-api" Nov 02 23:30:56.252596 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.252596 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="30477205-05db-46c2-b474-efe14b4ecdfd" containerName="cleanup" Nov 02 23:30:56.252951 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.252603 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="a463aa29-3e49-4bde-a4d2-868c217831c6" containerName="cleanup" Nov 02 23:30:56.252951 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.252611 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="30477205-05db-46c2-b474-efe14b4ecdfd" containerName="cleanup" Nov 02 23:30:56.257581 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.257564 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:30:56.261259 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.261231 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:30:56.261375 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.261264 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:30:56.261375 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.261282 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/465b7605-8ea0-42da-a879-d129b29ddf3f-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:30:56.261375 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.261346 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:30:56.261543 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.261385 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:30:56.261543 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.261474 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c69vd\" (UniqueName: \"kubernetes.io/projected/465b7605-8ea0-42da-a879-d129b29ddf3f-kube-api-access-c69vd\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:30:56.261543 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.261505 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:30:56.261701 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.261547 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/465b7605-8ea0-42da-a879-d129b29ddf3f-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:30:56.261701 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.261632 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/465b7605-8ea0-42da-a879-d129b29ddf3f-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:30:56.362299 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.362254 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c69vd\" (UniqueName: \"kubernetes.io/projected/465b7605-8ea0-42da-a879-d129b29ddf3f-kube-api-access-c69vd\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:30:56.362299 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.362299 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:30:56.362562 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.362499 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/465b7605-8ea0-42da-a879-d129b29ddf3f-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:30:56.362636 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.362617 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/465b7605-8ea0-42da-a879-d129b29ddf3f-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:30:56.362704 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.362684 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:30:56.362763 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.362713 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:30:56.362763 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.362729 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz"] Nov 02 23:30:56.362763 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.362740 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/465b7605-8ea0-42da-a879-d129b29ddf3f-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:30:56.362933 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.362768 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:30:56.362933 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.362816 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:30:56.363187 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.363164 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:30:56.363266 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.363219 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:30:56.363340 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.363294 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:30:56.363519 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.363496 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:30:56.363589 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.363543 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/465b7605-8ea0-42da-a879-d129b29ddf3f-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:30:56.364986 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.364956 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:30:56.365608 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.365589 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/465b7605-8ea0-42da-a879-d129b29ddf3f-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:30:56.423544 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.423516 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/465b7605-8ea0-42da-a879-d129b29ddf3f-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:30:56.423844 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.423815 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c69vd\" (UniqueName: \"kubernetes.io/projected/465b7605-8ea0-42da-a879-d129b29ddf3f-kube-api-access-c69vd\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:30:56.571676 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:56.571642 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:30:57.001089 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:57.001061 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz"] Nov 02 23:30:57.002677 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:30:57.002651 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod465b7605_8ea0_42da_a879_d129b29ddf3f.slice/crio-5dd3b3b3647f0889626544570d101975ce1a38de4fb4d9669ac9bb3a913a63aa WatchSource:0}: Error finding container 5dd3b3b3647f0889626544570d101975ce1a38de4fb4d9669ac9bb3a913a63aa: Status 404 returned error can't find the container with id 5dd3b3b3647f0889626544570d101975ce1a38de4fb4d9669ac9bb3a913a63aa Nov 02 23:30:57.005258 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:57.005225 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:30:57.005350 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:57.005290 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:30:57.005350 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:57.005318 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:30:57.566614 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:57.566568 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" event={"ID":"465b7605-8ea0-42da-a879-d129b29ddf3f","Type":"ContainerStarted","Data":"ce0ad02656d87501ef9ebe95999a783f9ba33258fe7cb52ed9eeb9e8e6cb185b"} Nov 02 23:30:57.567287 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:57.567261 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" event={"ID":"465b7605-8ea0-42da-a879-d129b29ddf3f","Type":"ContainerStarted","Data":"5dd3b3b3647f0889626544570d101975ce1a38de4fb4d9669ac9bb3a913a63aa"} Nov 02 23:30:57.571793 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:57.571763 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:30:57.702712 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:57.702653 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" podStartSLOduration=1.702638774 podStartE2EDuration="1.702638774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:57.701731939 +0000 UTC m=+2543.940229619" watchObservedRunningTime="2026-11-02 23:30:57.702638774 +0000 UTC m=+2543.941136437" Nov 02 23:30:57.847482 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:30:57.847390 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:30:58.582154 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:58.581295 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 02 23:30:58.582154 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:58.581360 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" podUID="465b7605-8ea0-42da-a879-d129b29ddf3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 02 23:30:59.575517 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:59.575471 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 02 23:30:59.575838 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:30:59.575536 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" podUID="465b7605-8ea0-42da-a879-d129b29ddf3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 02 23:31:00.574152 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:00.573454 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 02 23:31:00.574152 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:00.573515 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" podUID="465b7605-8ea0-42da-a879-d129b29ddf3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 02 23:31:01.575152 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:01.574787 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 02 23:31:01.575152 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:01.574869 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" podUID="465b7605-8ea0-42da-a879-d129b29ddf3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 02 23:31:02.576152 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:02.572483 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 02 23:31:02.576152 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:02.572555 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" podUID="465b7605-8ea0-42da-a879-d129b29ddf3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 02 23:31:03.572323 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:03.572268 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 02 23:31:03.572557 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:03.572342 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" podUID="465b7605-8ea0-42da-a879-d129b29ddf3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 02 23:31:04.577180 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:04.577105 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:31:04.577635 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:04.577514 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:31:04.578377 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:04.578357 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:31:09.906688 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:09.906657 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-adfeab-6f58b64699-msrnr"] Nov 02 23:31:10.024119 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:10.024070 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-adfeab-54956564cf-wqlzh"] Nov 02 23:31:14.907988 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:14.907935 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-adfeab-6f58b64699-msrnr" podUID="1a7ee0cb-5910-4811-8a00-df5d062ca16c" containerName="payload-processing" containerID="cri-o://9b7d5787b990e556d9ccdb22274249881961bc3936911db154dcd459af718b4d" gracePeriod=25 Nov 02 23:31:15.024815 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:15.024760 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-adfeab-54956564cf-wqlzh" podUID="89cc96f9-f51b-4132-b2dc-3a926896a29f" containerName="payload-pre-processing" containerID="cri-o://c7fa1d47c78fef51d898ba600b96bceb24b2c67432f23215da696e8daeeb566a" gracePeriod=25 Nov 02 23:31:15.232766 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:15.232735 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-adfeab-6f58b64699-msrnr" Nov 02 23:31:15.302545 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:15.302516 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-adfeab-54956564cf-wqlzh" Nov 02 23:31:15.345951 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:15.345921 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a7ee0cb-5910-4811-8a00-df5d062ca16c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7ee0cb-5910-4811-8a00-df5d062ca16c-plugins-config-volume\") pod \"1a7ee0cb-5910-4811-8a00-df5d062ca16c\" (UID: \"1a7ee0cb-5910-4811-8a00-df5d062ca16c\") " Nov 02 23:31:15.346110 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:15.346055 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a7ee0cb-5910-4811-8a00-df5d062ca16c-kube-api-access-pl8v4\" (UniqueName: \"kubernetes.io/projected/1a7ee0cb-5910-4811-8a00-df5d062ca16c-kube-api-access-pl8v4\") pod \"1a7ee0cb-5910-4811-8a00-df5d062ca16c\" (UID: \"1a7ee0cb-5910-4811-8a00-df5d062ca16c\") " Nov 02 23:31:15.346401 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:15.346377 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a7ee0cb-5910-4811-8a00-df5d062ca16c-plugins-config-volume" pod "1a7ee0cb-5910-4811-8a00-df5d062ca16c" (UID: "1a7ee0cb-5910-4811-8a00-df5d062ca16c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:31:15.346601 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:15.346587 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7ee0cb-5910-4811-8a00-df5d062ca16c-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:15.348762 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:15.348720 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7ee0cb-5910-4811-8a00-df5d062ca16c-kube-api-access-pl8v4" pod "1a7ee0cb-5910-4811-8a00-df5d062ca16c" (UID: "1a7ee0cb-5910-4811-8a00-df5d062ca16c"). InnerVolumeSpecName "kube-api-access-pl8v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:31:15.447865 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:15.447777 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89cc96f9-f51b-4132-b2dc-3a926896a29f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89cc96f9-f51b-4132-b2dc-3a926896a29f-plugins-config-volume\") pod \"89cc96f9-f51b-4132-b2dc-3a926896a29f\" (UID: \"89cc96f9-f51b-4132-b2dc-3a926896a29f\") " Nov 02 23:31:15.448101 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:15.448082 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89cc96f9-f51b-4132-b2dc-3a926896a29f-kube-api-access-c2h8n\" (UniqueName: \"kubernetes.io/projected/89cc96f9-f51b-4132-b2dc-3a926896a29f-kube-api-access-c2h8n\") pod \"89cc96f9-f51b-4132-b2dc-3a926896a29f\" (UID: \"89cc96f9-f51b-4132-b2dc-3a926896a29f\") " Nov 02 23:31:15.448229 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:15.448118 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89cc96f9-f51b-4132-b2dc-3a926896a29f-plugins-config-volume" pod "89cc96f9-f51b-4132-b2dc-3a926896a29f" (UID: "89cc96f9-f51b-4132-b2dc-3a926896a29f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:31:15.448516 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:15.448496 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl8v4\" (UniqueName: \"kubernetes.io/projected/1a7ee0cb-5910-4811-8a00-df5d062ca16c-kube-api-access-pl8v4\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:15.448589 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:15.448521 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89cc96f9-f51b-4132-b2dc-3a926896a29f-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:15.450394 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:15.450370 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89cc96f9-f51b-4132-b2dc-3a926896a29f-kube-api-access-c2h8n" pod "89cc96f9-f51b-4132-b2dc-3a926896a29f" (UID: "89cc96f9-f51b-4132-b2dc-3a926896a29f"). InnerVolumeSpecName "kube-api-access-c2h8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:31:15.549573 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:15.549544 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2h8n\" (UniqueName: \"kubernetes.io/projected/89cc96f9-f51b-4132-b2dc-3a926896a29f-kube-api-access-c2h8n\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:15.773752 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:15.773662 5727 generic.go:356] "Generic (PLEG): container finished" podID="89cc96f9-f51b-4132-b2dc-3a926896a29f" containerID="c7fa1d47c78fef51d898ba600b96bceb24b2c67432f23215da696e8daeeb566a" exitCode=0 Nov 02 23:31:15.773752 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:15.773736 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-adfeab-54956564cf-wqlzh" Nov 02 23:31:15.774004 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:15.773754 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-adfeab-54956564cf-wqlzh" event={"ID":"89cc96f9-f51b-4132-b2dc-3a926896a29f","Type":"ContainerDied","Data":"c7fa1d47c78fef51d898ba600b96bceb24b2c67432f23215da696e8daeeb566a"} Nov 02 23:31:15.774004 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:15.773818 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-adfeab-54956564cf-wqlzh" event={"ID":"89cc96f9-f51b-4132-b2dc-3a926896a29f","Type":"ContainerDied","Data":"f6ddf6052f1908b83d567f0adb4eff87ec7244103a5760faba4419af080badac"} Nov 02 23:31:15.774004 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:15.773848 5727 scope.go:122] "RemoveContainer" containerID="c7fa1d47c78fef51d898ba600b96bceb24b2c67432f23215da696e8daeeb566a" Nov 02 23:31:15.776194 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:15.776070 5727 generic.go:356] "Generic (PLEG): container finished" podID="1a7ee0cb-5910-4811-8a00-df5d062ca16c" containerID="9b7d5787b990e556d9ccdb22274249881961bc3936911db154dcd459af718b4d" exitCode=0 Nov 02 23:31:15.776305 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:15.776284 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-adfeab-6f58b64699-msrnr" event={"ID":"1a7ee0cb-5910-4811-8a00-df5d062ca16c","Type":"ContainerDied","Data":"9b7d5787b990e556d9ccdb22274249881961bc3936911db154dcd459af718b4d"} Nov 02 23:31:15.776389 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:15.776335 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-adfeab-6f58b64699-msrnr" event={"ID":"1a7ee0cb-5910-4811-8a00-df5d062ca16c","Type":"ContainerDied","Data":"b893148f312334a19809a5ad9eacc7ec1557a93415a6039329c53b3db12ea8df"} Nov 02 23:31:15.776458 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:15.776404 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-adfeab-6f58b64699-msrnr" Nov 02 23:31:15.789861 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:15.789804 5727 scope.go:122] "RemoveContainer" containerID="c7fa1d47c78fef51d898ba600b96bceb24b2c67432f23215da696e8daeeb566a" Nov 02 23:31:15.790200 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:31:15.790168 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7fa1d47c78fef51d898ba600b96bceb24b2c67432f23215da696e8daeeb566a\": container with ID starting with c7fa1d47c78fef51d898ba600b96bceb24b2c67432f23215da696e8daeeb566a not found: ID does not exist" containerID="c7fa1d47c78fef51d898ba600b96bceb24b2c67432f23215da696e8daeeb566a" Nov 02 23:31:15.790360 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:15.790206 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7fa1d47c78fef51d898ba600b96bceb24b2c67432f23215da696e8daeeb566a"} err="failed to get container status \"c7fa1d47c78fef51d898ba600b96bceb24b2c67432f23215da696e8daeeb566a\": rpc error: code = NotFound desc = could not find container \"c7fa1d47c78fef51d898ba600b96bceb24b2c67432f23215da696e8daeeb566a\": container with ID starting with c7fa1d47c78fef51d898ba600b96bceb24b2c67432f23215da696e8daeeb566a not found: ID does not exist" Nov 02 23:31:15.790360 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:15.790225 5727 scope.go:122] "RemoveContainer" containerID="9b7d5787b990e556d9ccdb22274249881961bc3936911db154dcd459af718b4d" Nov 02 23:31:15.800140 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:15.800104 5727 scope.go:122] "RemoveContainer" containerID="9b7d5787b990e556d9ccdb22274249881961bc3936911db154dcd459af718b4d" Nov 02 23:31:15.800404 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:31:15.800382 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b7d5787b990e556d9ccdb22274249881961bc3936911db154dcd459af718b4d\": container with ID starting with 9b7d5787b990e556d9ccdb22274249881961bc3936911db154dcd459af718b4d not found: ID does not exist" containerID="9b7d5787b990e556d9ccdb22274249881961bc3936911db154dcd459af718b4d" Nov 02 23:31:15.800479 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:15.800414 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b7d5787b990e556d9ccdb22274249881961bc3936911db154dcd459af718b4d"} err="failed to get container status \"9b7d5787b990e556d9ccdb22274249881961bc3936911db154dcd459af718b4d\": rpc error: code = NotFound desc = could not find container \"9b7d5787b990e556d9ccdb22274249881961bc3936911db154dcd459af718b4d\": container with ID starting with 9b7d5787b990e556d9ccdb22274249881961bc3936911db154dcd459af718b4d not found: ID does not exist" Nov 02 23:31:15.945945 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:15.945901 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-adfeab-54956564cf-wqlzh"] Nov 02 23:31:15.949820 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:15.949793 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-adfeab-54956564cf-wqlzh"] Nov 02 23:31:16.065509 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:16.065473 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-adfeab-6f58b64699-msrnr"] Nov 02 23:31:16.070592 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:16.070558 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-adfeab-6f58b64699-msrnr"] Nov 02 23:31:17.886920 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:17.886880 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a7ee0cb-5910-4811-8a00-df5d062ca16c" path="/var/lib/kubelet/pods/1a7ee0cb-5910-4811-8a00-df5d062ca16c/volumes" Nov 02 23:31:17.887527 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:17.887506 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89cc96f9-f51b-4132-b2dc-3a926896a29f" path="/var/lib/kubelet/pods/89cc96f9-f51b-4132-b2dc-3a926896a29f/volumes" Nov 02 23:31:26.486508 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.486468 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2"] Nov 02 23:31:26.487049 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.487033 5727 cpu_manager.go:414] "containerMap: removing container" podUID="a463aa29-3e49-4bde-a4d2-868c217831c6" containerName="cleanup" Nov 02 23:31:26.487111 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.487049 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a463aa29-3e49-4bde-a4d2-868c217831c6" containerName="cleanup" Nov 02 23:31:26.487111 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.487069 5727 cpu_manager.go:414] "containerMap: removing container" podUID="30477205-05db-46c2-b474-efe14b4ecdfd" containerName="cleanup" Nov 02 23:31:26.487111 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.487075 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30477205-05db-46c2-b474-efe14b4ecdfd" containerName="cleanup" Nov 02 23:31:26.487111 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.487104 5727 cpu_manager.go:414] "containerMap: removing container" podUID="1a7ee0cb-5910-4811-8a00-df5d062ca16c" containerName="payload-processing" Nov 02 23:31:26.487111 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.487110 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a7ee0cb-5910-4811-8a00-df5d062ca16c" containerName="payload-processing" Nov 02 23:31:26.487389 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.487144 5727 cpu_manager.go:414] "containerMap: removing container" podUID="89cc96f9-f51b-4132-b2dc-3a926896a29f" containerName="payload-pre-processing" Nov 02 23:31:26.487389 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.487149 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89cc96f9-f51b-4132-b2dc-3a926896a29f" containerName="payload-pre-processing" Nov 02 23:31:26.487389 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.487247 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a7ee0cb-5910-4811-8a00-df5d062ca16c" containerName="payload-processing" Nov 02 23:31:26.487389 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.487259 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="89cc96f9-f51b-4132-b2dc-3a926896a29f" containerName="payload-pre-processing" Nov 02 23:31:26.487389 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.487266 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="a463aa29-3e49-4bde-a4d2-868c217831c6" containerName="cleanup" Nov 02 23:31:26.492235 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.492215 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:26.586334 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.586304 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2"] Nov 02 23:31:26.586551 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.586530 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:26.586750 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.586729 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtfz6\" (UniqueName: \"kubernetes.io/projected/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-kube-api-access-jtfz6\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:26.586913 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.586885 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:26.587024 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.586981 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:26.587088 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.587029 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:26.587189 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.587099 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:26.587189 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.587175 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:26.587313 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.587213 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:26.587313 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.587258 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:26.631317 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.631292 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ec2f9a-openshift-default-dockercfg-4nt5j\"" Nov 02 23:31:26.688853 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.688822 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:26.689011 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.688873 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:26.689011 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.688909 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:26.689011 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.688942 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:26.689011 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.688962 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:26.689011 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.688989 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtfz6\" (UniqueName: \"kubernetes.io/projected/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-kube-api-access-jtfz6\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:26.689374 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.689015 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:26.689374 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.689156 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:26.689374 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.689209 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:26.689374 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.689281 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:26.689586 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.689369 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:26.689586 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.689570 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:26.689753 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.689724 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:26.689861 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.689725 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:26.691667 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.691647 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:26.691924 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.691902 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:26.749598 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.749524 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:26.749738 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.749700 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtfz6\" (UniqueName: \"kubernetes.io/projected/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-kube-api-access-jtfz6\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:26.809730 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:26.809695 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:27.252766 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:27.252736 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2"] Nov 02 23:31:27.254053 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:31:27.254025 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcabff31e_d3dd_469a_8f1d_f4e8fd22fe18.slice/crio-c466675f3535f18a63599d1afcf1ba9a075b3d2f59cd3fe7c17f52edecae27a7 WatchSource:0}: Error finding container c466675f3535f18a63599d1afcf1ba9a075b3d2f59cd3fe7c17f52edecae27a7: Status 404 returned error can't find the container with id c466675f3535f18a63599d1afcf1ba9a075b3d2f59cd3fe7c17f52edecae27a7 Nov 02 23:31:27.256606 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:27.256572 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:31:27.256697 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:27.256638 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:31:27.256697 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:27.256666 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:31:27.848077 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:31:27.848035 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:31:27.896394 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:27.896356 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" event={"ID":"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18","Type":"ContainerStarted","Data":"470c9d03cb4a9a50f315a394a7b662107a6ad1cda9b858eb14328283fb826a7b"} Nov 02 23:31:27.896572 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:27.896426 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" event={"ID":"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18","Type":"ContainerStarted","Data":"c466675f3535f18a63599d1afcf1ba9a075b3d2f59cd3fe7c17f52edecae27a7"} Nov 02 23:31:28.010622 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:28.010568 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" podStartSLOduration=2.010552764 podStartE2EDuration="2.010552764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:31:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:31:28.007210698 +0000 UTC m=+2574.245708371" watchObservedRunningTime="2026-11-02 23:31:28.010552764 +0000 UTC m=+2574.249050425" Nov 02 23:31:28.810368 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:28.810327 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:28.812426 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:28.812401 5727 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 02 23:31:28.812546 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:28.812457 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" podUID="cabff31e-d3dd-469a-8f1d-f4e8fd22fe18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 02 23:31:29.810525 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:29.810484 5727 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 02 23:31:29.810930 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:29.810552 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" podUID="cabff31e-d3dd-469a-8f1d-f4e8fd22fe18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 02 23:31:30.810224 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:30.810189 5727 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 02 23:31:30.810479 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:30.810252 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" podUID="cabff31e-d3dd-469a-8f1d-f4e8fd22fe18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 02 23:31:31.810424 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:31.810386 5727 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 02 23:31:31.810847 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:31.810442 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" podUID="cabff31e-d3dd-469a-8f1d-f4e8fd22fe18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 02 23:31:32.810617 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:32.810575 5727 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 02 23:31:32.811030 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:32.810652 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" podUID="cabff31e-d3dd-469a-8f1d-f4e8fd22fe18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 02 23:31:33.810028 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:33.809993 5727 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 02 23:31:33.810241 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:33.810065 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" podUID="cabff31e-d3dd-469a-8f1d-f4e8fd22fe18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 02 23:31:34.814696 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:34.814667 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:34.815082 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:34.814997 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:34.815629 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:34.815609 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:31:56.256466 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:56.256426 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj"] Nov 02 23:31:56.260388 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:56.260365 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:31:56.327860 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:56.327828 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:31:56.328103 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:56.328079 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac783482-b145-4065-b21a-0f54394f56c7-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:31:56.328209 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:56.328156 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:31:56.328268 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:56.328252 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac783482-b145-4065-b21a-0f54394f56c7-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:31:56.328312 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:56.328291 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:31:56.328368 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:56.328342 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:31:56.328448 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:56.328428 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac783482-b145-4065-b21a-0f54394f56c7-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:31:56.328502 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:56.328473 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:31:56.328502 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:56.328491 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb59s\" (UniqueName: \"kubernetes.io/projected/ac783482-b145-4065-b21a-0f54394f56c7-kube-api-access-xb59s\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:31:56.429857 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:56.429825 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac783482-b145-4065-b21a-0f54394f56c7-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:31:56.430032 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:56.429885 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:31:56.430032 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:56.429949 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac783482-b145-4065-b21a-0f54394f56c7-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:31:56.430032 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:56.429983 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:31:56.430032 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:56.430011 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:31:56.430311 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:56.430072 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac783482-b145-4065-b21a-0f54394f56c7-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:31:56.430311 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:56.430109 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:31:56.430311 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:56.430150 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb59s\" (UniqueName: \"kubernetes.io/projected/ac783482-b145-4065-b21a-0f54394f56c7-kube-api-access-xb59s\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:31:56.430311 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:56.430273 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:31:56.430720 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:56.430700 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:31:56.432542 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:56.431070 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:31:56.432542 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:56.431078 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:31:56.432542 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:56.431657 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:31:56.432542 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:56.432260 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac783482-b145-4065-b21a-0f54394f56c7-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:31:56.438148 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:56.434640 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac783482-b145-4065-b21a-0f54394f56c7-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:31:56.438517 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:56.438493 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:31:56.462590 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:56.462557 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj"] Nov 02 23:31:56.491670 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:56.491636 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac783482-b145-4065-b21a-0f54394f56c7-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:31:56.493348 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:56.493320 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb59s\" (UniqueName: \"kubernetes.io/projected/ac783482-b145-4065-b21a-0f54394f56c7-kube-api-access-xb59s\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:31:56.574207 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:56.574180 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:31:57.063074 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:57.063044 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj"] Nov 02 23:31:57.065536 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:31:57.065502 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac783482_b145_4065_b21a_0f54394f56c7.slice/crio-255ea192e78126c3ceb584caec9f6266afdbe0ca714857a3b491d807b4ff8960 WatchSource:0}: Error finding container 255ea192e78126c3ceb584caec9f6266afdbe0ca714857a3b491d807b4ff8960: Status 404 returned error can't find the container with id 255ea192e78126c3ceb584caec9f6266afdbe0ca714857a3b491d807b4ff8960 Nov 02 23:31:57.068307 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:57.068270 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:31:57.068392 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:57.068343 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:31:57.068392 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:57.068376 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:31:57.146931 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:57.146893 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" event={"ID":"ac783482-b145-4065-b21a-0f54394f56c7","Type":"ContainerStarted","Data":"255ea192e78126c3ceb584caec9f6266afdbe0ca714857a3b491d807b4ff8960"} Nov 02 23:31:57.849050 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:31:57.849014 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:31:58.173323 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:58.173223 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" event={"ID":"ac783482-b145-4065-b21a-0f54394f56c7","Type":"ContainerStarted","Data":"c5e64a7d7f3a3a4b3bf752c1e5e9d28a5ff24154c75640675a1500243d4dc7ad"} Nov 02 23:31:58.580529 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:58.580484 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:31:58.580919 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:58.580880 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 02 23:31:58.581112 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:58.581088 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" podUID="ac783482-b145-4065-b21a-0f54394f56c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 02 23:31:59.575541 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:59.575498 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 02 23:31:59.576074 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:31:59.575567 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" podUID="ac783482-b145-4065-b21a-0f54394f56c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 02 23:32:00.576153 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:00.575795 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 02 23:32:00.576153 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:00.575856 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" podUID="ac783482-b145-4065-b21a-0f54394f56c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 02 23:32:01.575477 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:01.575433 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 02 23:32:01.588289 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:01.588243 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" podUID="ac783482-b145-4065-b21a-0f54394f56c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 02 23:32:02.583157 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:02.574552 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 02 23:32:02.583157 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:02.574624 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" podUID="ac783482-b145-4065-b21a-0f54394f56c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 02 23:32:03.575293 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:03.575259 5727 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 02 23:32:03.575792 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:03.575329 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" podUID="ac783482-b145-4065-b21a-0f54394f56c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 02 23:32:04.582782 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:04.582742 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:32:04.583333 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:04.583103 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:32:04.584089 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:04.584057 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:32:04.758175 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:04.758075 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" podStartSLOduration=8.758055452 podStartE2EDuration="8.758055452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:31:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:31:58.405020736 +0000 UTC m=+2604.643518398" watchObservedRunningTime="2026-11-02 23:32:04.758055452 +0000 UTC m=+2610.996553115" Nov 02 23:32:17.021515 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:17.021430 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp"] Nov 02 23:32:17.021958 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:17.021640 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp" podUID="60f51458-ff9e-4b5d-97b1-f9f4ef10f580" containerName="maas-api" containerID="cri-o://6bbb47ab9c9901d07f8af644861012b1d52312d3154cea2a7ad31bbd52be90e7" gracePeriod=30 Nov 02 23:32:17.307059 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:17.307033 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp" Nov 02 23:32:17.356611 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:17.356575 5727 generic.go:356] "Generic (PLEG): container finished" podID="60f51458-ff9e-4b5d-97b1-f9f4ef10f580" containerID="6bbb47ab9c9901d07f8af644861012b1d52312d3154cea2a7ad31bbd52be90e7" exitCode=0 Nov 02 23:32:17.356783 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:17.356627 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp" Nov 02 23:32:17.356783 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:17.356660 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp" event={"ID":"60f51458-ff9e-4b5d-97b1-f9f4ef10f580","Type":"ContainerDied","Data":"6bbb47ab9c9901d07f8af644861012b1d52312d3154cea2a7ad31bbd52be90e7"} Nov 02 23:32:17.356783 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:17.356759 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp" event={"ID":"60f51458-ff9e-4b5d-97b1-f9f4ef10f580","Type":"ContainerDied","Data":"9839432a391831ad47ac51a3b134bbd1e888a63f8bf8210242fa0dbe5b106721"} Nov 02 23:32:17.356964 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:17.356790 5727 scope.go:122] "RemoveContainer" containerID="6bbb47ab9c9901d07f8af644861012b1d52312d3154cea2a7ad31bbd52be90e7" Nov 02 23:32:17.363208 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:17.363183 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60f51458-ff9e-4b5d-97b1-f9f4ef10f580-metrics-certs\" (UniqueName: \"kubernetes.io/secret/60f51458-ff9e-4b5d-97b1-f9f4ef10f580-metrics-certs\") pod \"60f51458-ff9e-4b5d-97b1-f9f4ef10f580\" (UID: \"60f51458-ff9e-4b5d-97b1-f9f4ef10f580\") " Nov 02 23:32:17.363316 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:17.363299 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60f51458-ff9e-4b5d-97b1-f9f4ef10f580-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60f51458-ff9e-4b5d-97b1-f9f4ef10f580-openshift-service-ca\") pod \"60f51458-ff9e-4b5d-97b1-f9f4ef10f580\" (UID: \"60f51458-ff9e-4b5d-97b1-f9f4ef10f580\") " Nov 02 23:32:17.363472 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:17.363449 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60f51458-ff9e-4b5d-97b1-f9f4ef10f580-kube-api-access-fwvzl\" (UniqueName: \"kubernetes.io/projected/60f51458-ff9e-4b5d-97b1-f9f4ef10f580-kube-api-access-fwvzl\") pod \"60f51458-ff9e-4b5d-97b1-f9f4ef10f580\" (UID: \"60f51458-ff9e-4b5d-97b1-f9f4ef10f580\") " Nov 02 23:32:17.363578 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:17.363554 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60f51458-ff9e-4b5d-97b1-f9f4ef10f580-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60f51458-ff9e-4b5d-97b1-f9f4ef10f580-maas-api-tls\") pod \"60f51458-ff9e-4b5d-97b1-f9f4ef10f580\" (UID: \"60f51458-ff9e-4b5d-97b1-f9f4ef10f580\") " Nov 02 23:32:17.363710 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:17.363685 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f51458-ff9e-4b5d-97b1-f9f4ef10f580-openshift-service-ca" pod "60f51458-ff9e-4b5d-97b1-f9f4ef10f580" (UID: "60f51458-ff9e-4b5d-97b1-f9f4ef10f580"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:32:17.364265 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:17.364244 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60f51458-ff9e-4b5d-97b1-f9f4ef10f580-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:17.365669 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:17.365633 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f51458-ff9e-4b5d-97b1-f9f4ef10f580-metrics-certs" pod "60f51458-ff9e-4b5d-97b1-f9f4ef10f580" (UID: "60f51458-ff9e-4b5d-97b1-f9f4ef10f580"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:32:17.365746 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:17.365731 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60f51458-ff9e-4b5d-97b1-f9f4ef10f580-kube-api-access-fwvzl" pod "60f51458-ff9e-4b5d-97b1-f9f4ef10f580" (UID: "60f51458-ff9e-4b5d-97b1-f9f4ef10f580"). InnerVolumeSpecName "kube-api-access-fwvzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:32:17.367266 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:17.367236 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f51458-ff9e-4b5d-97b1-f9f4ef10f580-maas-api-tls" pod "60f51458-ff9e-4b5d-97b1-f9f4ef10f580" (UID: "60f51458-ff9e-4b5d-97b1-f9f4ef10f580"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:32:17.368096 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:17.368080 5727 scope.go:122] "RemoveContainer" containerID="6bbb47ab9c9901d07f8af644861012b1d52312d3154cea2a7ad31bbd52be90e7" Nov 02 23:32:17.368414 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:32:17.368396 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bbb47ab9c9901d07f8af644861012b1d52312d3154cea2a7ad31bbd52be90e7\": container with ID starting with 6bbb47ab9c9901d07f8af644861012b1d52312d3154cea2a7ad31bbd52be90e7 not found: ID does not exist" containerID="6bbb47ab9c9901d07f8af644861012b1d52312d3154cea2a7ad31bbd52be90e7" Nov 02 23:32:17.368483 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:17.368427 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bbb47ab9c9901d07f8af644861012b1d52312d3154cea2a7ad31bbd52be90e7"} err="failed to get container status \"6bbb47ab9c9901d07f8af644861012b1d52312d3154cea2a7ad31bbd52be90e7\": rpc error: code = NotFound desc = could not find container \"6bbb47ab9c9901d07f8af644861012b1d52312d3154cea2a7ad31bbd52be90e7\": container with ID starting with 6bbb47ab9c9901d07f8af644861012b1d52312d3154cea2a7ad31bbd52be90e7 not found: ID does not exist" Nov 02 23:32:17.465471 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:17.465438 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60f51458-ff9e-4b5d-97b1-f9f4ef10f580-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:17.465643 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:17.465476 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60f51458-ff9e-4b5d-97b1-f9f4ef10f580-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:17.465643 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:17.465494 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwvzl\" (UniqueName: \"kubernetes.io/projected/60f51458-ff9e-4b5d-97b1-f9f4ef10f580-kube-api-access-fwvzl\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:17.784330 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:17.784297 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp"] Nov 02 23:32:17.787340 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:17.787316 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-014ab4-68f68fdf-4mrfp"] Nov 02 23:32:17.883548 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:17.883518 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60f51458-ff9e-4b5d-97b1-f9f4ef10f580" path="/var/lib/kubelet/pods/60f51458-ff9e-4b5d-97b1-f9f4ef10f580/volumes" Nov 02 23:32:26.948712 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:26.948680 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9"] Nov 02 23:32:26.949328 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:26.949312 5727 cpu_manager.go:414] "containerMap: removing container" podUID="60f51458-ff9e-4b5d-97b1-f9f4ef10f580" containerName="maas-api" Nov 02 23:32:26.949378 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:26.949330 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60f51458-ff9e-4b5d-97b1-f9f4ef10f580" containerName="maas-api" Nov 02 23:32:26.949450 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:26.949441 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="60f51458-ff9e-4b5d-97b1-f9f4ef10f580" containerName="maas-api" Nov 02 23:32:26.954300 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:26.954279 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:27.091544 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.091505 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:27.091727 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.091645 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf35636b-a796-4dfc-83af-9f052a649ae3-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:27.091727 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.091684 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvmzv\" (UniqueName: \"kubernetes.io/projected/bf35636b-a796-4dfc-83af-9f052a649ae3-kube-api-access-vvmzv\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:27.091868 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.091836 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:27.091925 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.091898 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:27.091977 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.091925 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf35636b-a796-4dfc-83af-9f052a649ae3-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:27.092048 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.092032 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:27.092087 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.092072 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:27.092239 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.092213 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf35636b-a796-4dfc-83af-9f052a649ae3-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:27.151876 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.151846 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9"] Nov 02 23:32:27.193823 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.193794 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:27.193991 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.193839 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:27.193991 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.193858 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf35636b-a796-4dfc-83af-9f052a649ae3-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:27.193991 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.193900 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:27.193991 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.193926 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:27.193991 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.193972 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf35636b-a796-4dfc-83af-9f052a649ae3-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:27.194280 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.194044 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:27.194280 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.194103 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf35636b-a796-4dfc-83af-9f052a649ae3-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:27.194280 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.194145 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvmzv\" (UniqueName: \"kubernetes.io/projected/bf35636b-a796-4dfc-83af-9f052a649ae3-kube-api-access-vvmzv\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:27.194437 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.194284 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:27.194492 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.194429 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:27.194554 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.194532 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:27.194715 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.194650 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf35636b-a796-4dfc-83af-9f052a649ae3-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:27.195063 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.195042 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:27.196510 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.196485 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:27.197159 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.197119 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf35636b-a796-4dfc-83af-9f052a649ae3-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:27.254501 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.254448 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf35636b-a796-4dfc-83af-9f052a649ae3-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:27.254912 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.254895 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvmzv\" (UniqueName: \"kubernetes.io/projected/bf35636b-a796-4dfc-83af-9f052a649ae3-kube-api-access-vvmzv\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:27.268412 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.268387 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:27.707941 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.707910 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9"] Nov 02 23:32:27.708894 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:32:27.708864 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf35636b_a796_4dfc_83af_9f052a649ae3.slice/crio-56794ef6077543ded36d7388cc240a7fa62ebed410eb7ce7cc81f6a17ec02ef7 WatchSource:0}: Error finding container 56794ef6077543ded36d7388cc240a7fa62ebed410eb7ce7cc81f6a17ec02ef7: Status 404 returned error can't find the container with id 56794ef6077543ded36d7388cc240a7fa62ebed410eb7ce7cc81f6a17ec02ef7 Nov 02 23:32:27.711553 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.711522 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:32:27.711634 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.711582 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:32:27.711634 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:27.711611 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:32:27.853448 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:32:27.849319 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:32:28.447171 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:28.446247 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" event={"ID":"bf35636b-a796-4dfc-83af-9f052a649ae3","Type":"ContainerStarted","Data":"1c347628a4e5641dbf245979625e854d9238469958709bb5b38551e6c30f3672"} Nov 02 23:32:28.447171 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:28.446318 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" event={"ID":"bf35636b-a796-4dfc-83af-9f052a649ae3","Type":"ContainerStarted","Data":"56794ef6077543ded36d7388cc240a7fa62ebed410eb7ce7cc81f6a17ec02ef7"} Nov 02 23:32:28.691155 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:28.691050 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" podStartSLOduration=2.691028652 podStartE2EDuration="2.691028652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:32:28.617616113 +0000 UTC m=+2634.856113799" watchObservedRunningTime="2026-11-02 23:32:28.691028652 +0000 UTC m=+2634.929526315" Nov 02 23:32:29.270548 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:29.270510 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:29.285299 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:29.285262 5727 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 02 23:32:29.285570 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:29.285541 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" podUID="bf35636b-a796-4dfc-83af-9f052a649ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 02 23:32:30.273150 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:30.272197 5727 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 02 23:32:30.273150 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:30.272276 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" podUID="bf35636b-a796-4dfc-83af-9f052a649ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 02 23:32:31.278153 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:31.275500 5727 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 02 23:32:31.278153 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:31.275614 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" podUID="bf35636b-a796-4dfc-83af-9f052a649ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 02 23:32:32.270154 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:32.269478 5727 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 02 23:32:32.270154 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:32.269550 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" podUID="bf35636b-a796-4dfc-83af-9f052a649ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 02 23:32:33.272154 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:33.271474 5727 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 02 23:32:33.272154 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:33.271537 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" podUID="bf35636b-a796-4dfc-83af-9f052a649ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 02 23:32:34.268997 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:34.268953 5727 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 02 23:32:34.269275 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:34.269039 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" podUID="bf35636b-a796-4dfc-83af-9f052a649ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 02 23:32:36.291150 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:36.287222 5727 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:32:36.291150 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:36.287291 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" podUID="bf35636b-a796-4dfc-83af-9f052a649ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:32:36.542213 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:36.534762 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" podUID="cdc40e13-d236-4739-a4f5-098aea3fb308" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Nov 02 23:32:37.328287 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:37.328252 5727 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:32:37.328775 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:37.328325 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" podUID="bf35636b-a796-4dfc-83af-9f052a649ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:32:37.758236 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:37.758140 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:37.758589 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:37.758566 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:37.759789 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:37.759763 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:32:49.895340 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:49.895308 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs"] Nov 02 23:32:49.895832 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:49.895533 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" podUID="c2048b92-e5d0-4a0c-9951-893ecc868e9b" containerName="istio-proxy" containerID="cri-o://591e5bd02b8b9f7f9efd03a4be454b40b6746243e11d2dfc91a5c65734532d60" gracePeriod=30 Nov 02 23:32:51.627969 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:51.627934 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f46b61-9bf775457-hdjr6"] Nov 02 23:32:51.635585 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:51.635555 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f46b61-9bf775457-hdjr6" Nov 02 23:32:51.702021 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:51.701978 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f46b61-9bf775457-hdjr6"] Nov 02 23:32:51.718758 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:51.718260 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc2cc\" (UniqueName: \"kubernetes.io/projected/4b213bcf-90cf-414e-9432-e7887fa901fa-kube-api-access-wc2cc\") pod \"payload-processing-e2e-shared-a-w3-f46b61-9bf775457-hdjr6\" (UID: \"4b213bcf-90cf-414e-9432-e7887fa901fa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f46b61-9bf775457-hdjr6" Nov 02 23:32:51.718758 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:51.718490 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b213bcf-90cf-414e-9432-e7887fa901fa-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-f46b61-9bf775457-hdjr6\" (UID: \"4b213bcf-90cf-414e-9432-e7887fa901fa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f46b61-9bf775457-hdjr6" Nov 02 23:32:51.735989 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:51.735961 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-f46b61\"" Nov 02 23:32:51.736118 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:51.736051 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-f46b61-dockercfg-jsx6s\"" Nov 02 23:32:51.819829 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:51.819799 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b213bcf-90cf-414e-9432-e7887fa901fa-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-f46b61-9bf775457-hdjr6\" (UID: \"4b213bcf-90cf-414e-9432-e7887fa901fa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f46b61-9bf775457-hdjr6" Nov 02 23:32:51.819985 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:51.819881 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc2cc\" (UniqueName: \"kubernetes.io/projected/4b213bcf-90cf-414e-9432-e7887fa901fa-kube-api-access-wc2cc\") pod \"payload-processing-e2e-shared-a-w3-f46b61-9bf775457-hdjr6\" (UID: \"4b213bcf-90cf-414e-9432-e7887fa901fa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f46b61-9bf775457-hdjr6" Nov 02 23:32:51.820552 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:51.820530 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b213bcf-90cf-414e-9432-e7887fa901fa-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-f46b61-9bf775457-hdjr6\" (UID: \"4b213bcf-90cf-414e-9432-e7887fa901fa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f46b61-9bf775457-hdjr6" Nov 02 23:32:51.880903 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:51.880833 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc2cc\" (UniqueName: \"kubernetes.io/projected/4b213bcf-90cf-414e-9432-e7887fa901fa-kube-api-access-wc2cc\") pod \"payload-processing-e2e-shared-a-w3-f46b61-9bf775457-hdjr6\" (UID: \"4b213bcf-90cf-414e-9432-e7887fa901fa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f46b61-9bf775457-hdjr6" Nov 02 23:32:51.964408 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:51.964377 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f46b61-9bf775457-hdjr6" Nov 02 23:32:52.350073 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:52.350041 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f46b61-9bf775457-hdjr6"] Nov 02 23:32:53.008173 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:53.008106 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f46b61-9bf775457-hdjr6" event={"ID":"4b213bcf-90cf-414e-9432-e7887fa901fa","Type":"ContainerStarted","Data":"11be3f531df24d6c9014125f931cfd4fd7699bcbcec4b553bc78abd2e28ecd5b"} Nov 02 23:32:53.008578 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:53.008193 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f46b61-9bf775457-hdjr6" event={"ID":"4b213bcf-90cf-414e-9432-e7887fa901fa","Type":"ContainerStarted","Data":"a4bc5fd02895a8aaa8dd5ec34aa5f459fb54fa1130966edb417bb511b205e2b6"} Nov 02 23:32:53.008578 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:53.008225 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f46b61-9bf775457-hdjr6" Nov 02 23:32:53.124851 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:53.124805 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f46b61-9bf775457-hdjr6" podStartSLOduration=2.124791513 podStartE2EDuration="2.124791513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:32:53.122255525 +0000 UTC m=+2659.360753209" watchObservedRunningTime="2026-11-02 23:32:53.124791513 +0000 UTC m=+2659.363289173" Nov 02 23:32:55.023713 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:55.023678 5727 generic.go:356] "Generic (PLEG): container finished" podID="c2048b92-e5d0-4a0c-9951-893ecc868e9b" containerID="591e5bd02b8b9f7f9efd03a4be454b40b6746243e11d2dfc91a5c65734532d60" exitCode=0 Nov 02 23:32:55.024019 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:55.023745 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" event={"ID":"c2048b92-e5d0-4a0c-9951-893ecc868e9b","Type":"ContainerDied","Data":"591e5bd02b8b9f7f9efd03a4be454b40b6746243e11d2dfc91a5c65734532d60"} Nov 02 23:32:55.180768 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:55.180706 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:32:55.362392 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:55.362355 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istio-podinfo\") pod \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " Nov 02 23:32:55.362580 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:55.362416 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istio-token\" (UniqueName: \"kubernetes.io/projected/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istio-token\") pod \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " Nov 02 23:32:55.362580 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:55.362444 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istiod-ca-cert\") pod \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " Nov 02 23:32:55.362580 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:55.362474 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istio-envoy\") pod \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " Nov 02 23:32:55.362757 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:55.362585 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-credential-socket\") pod \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " Nov 02 23:32:55.362757 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:55.362610 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-workload-socket\") pod \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " Nov 02 23:32:55.362757 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:55.362672 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istio-data\") pod \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " Nov 02 23:32:55.362757 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:55.362729 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2048b92-e5d0-4a0c-9951-893ecc868e9b-kube-api-access-8dg22\" (UniqueName: \"kubernetes.io/projected/c2048b92-e5d0-4a0c-9951-893ecc868e9b-kube-api-access-8dg22\") pod \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " Nov 02 23:32:55.362964 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:55.362807 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-workload-certs\") pod \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\" (UID: \"c2048b92-e5d0-4a0c-9951-893ecc868e9b\") " Nov 02 23:32:55.363022 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:55.362992 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-workload-socket" pod "c2048b92-e5d0-4a0c-9951-893ecc868e9b" (UID: "c2048b92-e5d0-4a0c-9951-893ecc868e9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:32:55.363022 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:55.362914 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istiod-ca-cert" pod "c2048b92-e5d0-4a0c-9951-893ecc868e9b" (UID: "c2048b92-e5d0-4a0c-9951-893ecc868e9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:32:55.363249 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:55.363226 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istio-data" pod "c2048b92-e5d0-4a0c-9951-893ecc868e9b" (UID: "c2048b92-e5d0-4a0c-9951-893ecc868e9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:32:55.363458 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:55.363434 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-workload-certs" pod "c2048b92-e5d0-4a0c-9951-893ecc868e9b" (UID: "c2048b92-e5d0-4a0c-9951-893ecc868e9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:32:55.363630 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:55.363472 5727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istiod-ca-cert\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:55.363731 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:55.363716 5727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-workload-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:55.363812 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:55.363801 5727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istio-data\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:55.364099 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:55.364081 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-credential-socket" pod "c2048b92-e5d0-4a0c-9951-893ecc868e9b" (UID: "c2048b92-e5d0-4a0c-9951-893ecc868e9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:32:55.365958 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:55.365937 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istio-envoy" pod "c2048b92-e5d0-4a0c-9951-893ecc868e9b" (UID: "c2048b92-e5d0-4a0c-9951-893ecc868e9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:32:55.366097 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:55.365937 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istio-podinfo" pod "c2048b92-e5d0-4a0c-9951-893ecc868e9b" (UID: "c2048b92-e5d0-4a0c-9951-893ecc868e9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:32:55.366615 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:55.366585 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istio-token" pod "c2048b92-e5d0-4a0c-9951-893ecc868e9b" (UID: "c2048b92-e5d0-4a0c-9951-893ecc868e9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:32:55.367121 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:55.367097 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2048b92-e5d0-4a0c-9951-893ecc868e9b-kube-api-access-8dg22" pod "c2048b92-e5d0-4a0c-9951-893ecc868e9b" (UID: "c2048b92-e5d0-4a0c-9951-893ecc868e9b"). InnerVolumeSpecName "kube-api-access-8dg22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:32:55.465687 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:55.465594 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dg22\" (UniqueName: \"kubernetes.io/projected/c2048b92-e5d0-4a0c-9951-893ecc868e9b-kube-api-access-8dg22\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:55.465687 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:55.465627 5727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-workload-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:55.465687 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:55.465641 5727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istio-podinfo\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:55.465687 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:55.465653 5727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istio-token\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:55.465687 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:55.465665 5727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-istio-envoy\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:55.465687 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:55.465677 5727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2048b92-e5d0-4a0c-9951-893ecc868e9b-credential-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:56.031911 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:56.031877 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" Nov 02 23:32:56.032382 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:56.031905 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs" event={"ID":"c2048b92-e5d0-4a0c-9951-893ecc868e9b","Type":"ContainerDied","Data":"43d736d4d831242da173fe6d4f379eec5f5d2f28f908086f72cad50603b4328c"} Nov 02 23:32:56.032382 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:56.031980 5727 scope.go:122] "RemoveContainer" containerID="591e5bd02b8b9f7f9efd03a4be454b40b6746243e11d2dfc91a5c65734532d60" Nov 02 23:32:56.196958 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:56.196877 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs"] Nov 02 23:32:56.246565 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:56.246533 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-pnkcs"] Nov 02 23:32:57.203603 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.203570 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7"] Nov 02 23:32:57.204283 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.204268 5727 cpu_manager.go:414] "containerMap: removing container" podUID="c2048b92-e5d0-4a0c-9951-893ecc868e9b" containerName="istio-proxy" Nov 02 23:32:57.204283 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.204284 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2048b92-e5d0-4a0c-9951-893ecc868e9b" containerName="istio-proxy" Nov 02 23:32:57.204401 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.204391 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2048b92-e5d0-4a0c-9951-893ecc868e9b" containerName="istio-proxy" Nov 02 23:32:57.209480 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.209460 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:32:57.267749 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.267717 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7"] Nov 02 23:32:57.297704 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.297667 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:32:57.297879 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.297767 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:32:57.297879 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.297831 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:32:57.297879 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.297858 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5415246-c0df-4f8d-a382-34a674b1f21f-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:32:57.298070 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.297900 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5415246-c0df-4f8d-a382-34a674b1f21f-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:32:57.298070 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.297990 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5415246-c0df-4f8d-a382-34a674b1f21f-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:32:57.298070 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.298025 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:32:57.298273 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.298248 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:32:57.298340 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.298281 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbhjg\" (UniqueName: \"kubernetes.io/projected/e5415246-c0df-4f8d-a382-34a674b1f21f-kube-api-access-zbhjg\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:32:57.400102 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.400049 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:32:57.400320 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.400112 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:32:57.400320 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.400224 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:32:57.400320 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.400251 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5415246-c0df-4f8d-a382-34a674b1f21f-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:32:57.400320 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.400296 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5415246-c0df-4f8d-a382-34a674b1f21f-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:32:57.400581 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.400391 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5415246-c0df-4f8d-a382-34a674b1f21f-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:32:57.400581 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.400427 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:32:57.400581 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.400505 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:32:57.400581 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.400534 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbhjg\" (UniqueName: \"kubernetes.io/projected/e5415246-c0df-4f8d-a382-34a674b1f21f-kube-api-access-zbhjg\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:32:57.400793 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.400581 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:32:57.401028 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.401005 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:32:57.401658 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.401633 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:32:57.401842 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.401823 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:32:57.403419 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.403384 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5415246-c0df-4f8d-a382-34a674b1f21f-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:32:57.406104 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.404466 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5415246-c0df-4f8d-a382-34a674b1f21f-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:32:57.407268 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.406576 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:32:57.472200 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.469722 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbhjg\" (UniqueName: \"kubernetes.io/projected/e5415246-c0df-4f8d-a382-34a674b1f21f-kube-api-access-zbhjg\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:32:57.472200 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.471419 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5415246-c0df-4f8d-a382-34a674b1f21f-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:32:57.525262 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.525233 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:32:57.892180 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.891024 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2048b92-e5d0-4a0c-9951-893ecc868e9b" path="/var/lib/kubelet/pods/c2048b92-e5d0-4a0c-9951-893ecc868e9b/volumes" Nov 02 23:32:57.892180 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:32:57.891764 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:32:57.975674 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:57.975150 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7"] Nov 02 23:32:57.996780 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:32:57.991891 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5415246_c0df_4f8d_a382_34a674b1f21f.slice/crio-417b5ba77df971fbd7b85c5a633bd1bcea3e7c03b1662d8a1e0586fc45518a60 WatchSource:0}: Error finding container 417b5ba77df971fbd7b85c5a633bd1bcea3e7c03b1662d8a1e0586fc45518a60: Status 404 returned error can't find the container with id 417b5ba77df971fbd7b85c5a633bd1bcea3e7c03b1662d8a1e0586fc45518a60 Nov 02 23:32:58.007064 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:58.007016 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:32:58.007211 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:58.007116 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:32:58.007211 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:58.007183 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:32:58.057285 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:58.057178 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" event={"ID":"e5415246-c0df-4f8d-a382-34a674b1f21f","Type":"ContainerStarted","Data":"417b5ba77df971fbd7b85c5a633bd1bcea3e7c03b1662d8a1e0586fc45518a60"} Nov 02 23:32:59.049166 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:59.047214 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-440c60-65b68b4cd7-82p9f"] Nov 02 23:32:59.335592 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:59.335342 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-440c60-65b68b4cd7-82p9f"] Nov 02 23:32:59.335592 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:59.335479 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-440c60-65b68b4cd7-82p9f" Nov 02 23:32:59.412153 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:59.412096 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" event={"ID":"e5415246-c0df-4f8d-a382-34a674b1f21f","Type":"ContainerStarted","Data":"477dc86b29f9b9ea4eea1c7c2b1ad30cd7391097ae1eb7d600fbfda318afd05e"} Nov 02 23:32:59.436163 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:59.418621 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbsvb\" (UniqueName: \"kubernetes.io/projected/1d559e50-9c39-42dd-8ac1-6d42ebb726c4-kube-api-access-cbsvb\") pod \"payload-processing-e2e-shared-a-w0-440c60-65b68b4cd7-82p9f\" (UID: \"1d559e50-9c39-42dd-8ac1-6d42ebb726c4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-440c60-65b68b4cd7-82p9f" Nov 02 23:32:59.437829 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:59.437797 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d559e50-9c39-42dd-8ac1-6d42ebb726c4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-440c60-65b68b4cd7-82p9f\" (UID: \"1d559e50-9c39-42dd-8ac1-6d42ebb726c4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-440c60-65b68b4cd7-82p9f" Nov 02 23:32:59.438267 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:59.438239 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-440c60-dockercfg-skjr5\"" Nov 02 23:32:59.438373 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:59.438314 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-440c60\"" Nov 02 23:32:59.527917 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:59.527890 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:32:59.537718 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:59.531443 5727 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 02 23:32:59.537986 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:59.537962 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" podUID="e5415246-c0df-4f8d-a382-34a674b1f21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 02 23:32:59.648454 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:59.648319 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" podStartSLOduration=2.6482969990000003 podStartE2EDuration="2.648296999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:32:59.611979436 +0000 UTC m=+2665.850477099" watchObservedRunningTime="2026-11-02 23:32:59.648296999 +0000 UTC m=+2665.886794662" Nov 02 23:32:59.700366 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:59.700323 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d559e50-9c39-42dd-8ac1-6d42ebb726c4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-440c60-65b68b4cd7-82p9f\" (UID: \"1d559e50-9c39-42dd-8ac1-6d42ebb726c4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-440c60-65b68b4cd7-82p9f" Nov 02 23:32:59.700976 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:59.700919 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbsvb\" (UniqueName: \"kubernetes.io/projected/1d559e50-9c39-42dd-8ac1-6d42ebb726c4-kube-api-access-cbsvb\") pod \"payload-processing-e2e-shared-a-w0-440c60-65b68b4cd7-82p9f\" (UID: \"1d559e50-9c39-42dd-8ac1-6d42ebb726c4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-440c60-65b68b4cd7-82p9f" Nov 02 23:32:59.705916 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:59.703440 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d559e50-9c39-42dd-8ac1-6d42ebb726c4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-440c60-65b68b4cd7-82p9f\" (UID: \"1d559e50-9c39-42dd-8ac1-6d42ebb726c4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-440c60-65b68b4cd7-82p9f" Nov 02 23:32:59.790053 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:59.790010 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbsvb\" (UniqueName: \"kubernetes.io/projected/1d559e50-9c39-42dd-8ac1-6d42ebb726c4-kube-api-access-cbsvb\") pod \"payload-processing-e2e-shared-a-w0-440c60-65b68b4cd7-82p9f\" (UID: \"1d559e50-9c39-42dd-8ac1-6d42ebb726c4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-440c60-65b68b4cd7-82p9f" Nov 02 23:32:59.866462 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:32:59.865366 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-440c60-65b68b4cd7-82p9f" Nov 02 23:33:00.281694 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:00.281664 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-440c60-65b68b4cd7-82p9f"] Nov 02 23:33:00.421373 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:00.421340 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-440c60-65b68b4cd7-82p9f" event={"ID":"1d559e50-9c39-42dd-8ac1-6d42ebb726c4","Type":"ContainerStarted","Data":"5f0c6fee4d188c1fc478716bb4842808741dbe8d1145bdb8d9fbb3a12d4d6b48"} Nov 02 23:33:00.421536 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:00.421392 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-440c60-65b68b4cd7-82p9f" event={"ID":"1d559e50-9c39-42dd-8ac1-6d42ebb726c4","Type":"ContainerStarted","Data":"b8e2e3031d74be5e28b75357c96a10d1f0501305123888b41e4a7ebf54815c1b"} Nov 02 23:33:00.421588 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:00.421542 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-440c60-65b68b4cd7-82p9f" Nov 02 23:33:00.526504 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:00.526423 5727 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 02 23:33:00.526504 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:00.526469 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" podUID="e5415246-c0df-4f8d-a382-34a674b1f21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 02 23:33:00.536985 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:00.536931 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-440c60-65b68b4cd7-82p9f" podStartSLOduration=2.536914141 podStartE2EDuration="2.536914141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:33:00.535879965 +0000 UTC m=+2666.774377628" watchObservedRunningTime="2026-11-02 23:33:00.536914141 +0000 UTC m=+2666.775411804" Nov 02 23:33:01.526829 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:01.526792 5727 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 02 23:33:01.527260 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:01.526856 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" podUID="e5415246-c0df-4f8d-a382-34a674b1f21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 02 23:33:02.526060 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:02.526027 5727 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 02 23:33:02.526272 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:02.526077 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" podUID="e5415246-c0df-4f8d-a382-34a674b1f21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 02 23:33:03.525931 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:03.525899 5727 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 02 23:33:03.526341 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:03.525947 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" podUID="e5415246-c0df-4f8d-a382-34a674b1f21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 02 23:33:04.019064 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:04.019035 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f46b61-9bf775457-hdjr6" Nov 02 23:33:04.526074 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:04.526043 5727 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 02 23:33:04.526485 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:04.526155 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" podUID="e5415246-c0df-4f8d-a382-34a674b1f21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 02 23:33:05.933586 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:05.933533 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:33:05.934447 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:05.934427 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:33:05.935307 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:05.935277 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:33:11.451222 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:11.451195 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-440c60-65b68b4cd7-82p9f" Nov 02 23:33:21.849250 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:21.849220 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-980b82-84d578bd5f-hz792"] Nov 02 23:33:21.853845 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:21.853823 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-980b82-84d578bd5f-hz792" Nov 02 23:33:21.906043 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:21.906013 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-980b82-dockercfg-kcx4j\"" Nov 02 23:33:21.911288 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:21.911258 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-980b82-84d578bd5f-hz792"] Nov 02 23:33:21.919170 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:21.919141 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd7r9\" (UniqueName: \"kubernetes.io/projected/50c5d765-70d6-467e-9400-0a79f379131c-kube-api-access-xd7r9\") pod \"payload-pre-processing-e2e-shared-a-w2-980b82-84d578bd5f-hz792\" (UID: \"50c5d765-70d6-467e-9400-0a79f379131c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-980b82-84d578bd5f-hz792" Nov 02 23:33:21.919405 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:21.919383 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50c5d765-70d6-467e-9400-0a79f379131c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-980b82-84d578bd5f-hz792\" (UID: \"50c5d765-70d6-467e-9400-0a79f379131c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-980b82-84d578bd5f-hz792" Nov 02 23:33:22.008226 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:21.994786 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-980b82\"" Nov 02 23:33:22.020950 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:22.020914 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50c5d765-70d6-467e-9400-0a79f379131c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-980b82-84d578bd5f-hz792\" (UID: \"50c5d765-70d6-467e-9400-0a79f379131c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-980b82-84d578bd5f-hz792" Nov 02 23:33:22.022114 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:22.022091 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd7r9\" (UniqueName: \"kubernetes.io/projected/50c5d765-70d6-467e-9400-0a79f379131c-kube-api-access-xd7r9\") pod \"payload-pre-processing-e2e-shared-a-w2-980b82-84d578bd5f-hz792\" (UID: \"50c5d765-70d6-467e-9400-0a79f379131c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-980b82-84d578bd5f-hz792" Nov 02 23:33:22.022697 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:22.021947 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50c5d765-70d6-467e-9400-0a79f379131c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-980b82-84d578bd5f-hz792\" (UID: \"50c5d765-70d6-467e-9400-0a79f379131c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-980b82-84d578bd5f-hz792" Nov 02 23:33:22.097746 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:22.097710 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd7r9\" (UniqueName: \"kubernetes.io/projected/50c5d765-70d6-467e-9400-0a79f379131c-kube-api-access-xd7r9\") pod \"payload-pre-processing-e2e-shared-a-w2-980b82-84d578bd5f-hz792\" (UID: \"50c5d765-70d6-467e-9400-0a79f379131c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-980b82-84d578bd5f-hz792" Nov 02 23:33:22.169232 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:22.169119 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-980b82-84d578bd5f-hz792" Nov 02 23:33:22.548335 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:22.548307 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-980b82-84d578bd5f-hz792"] Nov 02 23:33:22.549169 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:33:22.549106 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c5d765_70d6_467e_9400_0a79f379131c.slice/crio-32945a776bb31af1b15408062a530f934f8884f1d3e0f4dfdc0a26fdd5c77883 WatchSource:0}: Error finding container 32945a776bb31af1b15408062a530f934f8884f1d3e0f4dfdc0a26fdd5c77883: Status 404 returned error can't find the container with id 32945a776bb31af1b15408062a530f934f8884f1d3e0f4dfdc0a26fdd5c77883 Nov 02 23:33:22.672382 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:22.672352 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-980b82-84d578bd5f-hz792" event={"ID":"50c5d765-70d6-467e-9400-0a79f379131c","Type":"ContainerStarted","Data":"5fcee09b0a56aeb327f2492b2e6edf12a6acedbe56bdb282953ed4aa04c733bc"} Nov 02 23:33:22.672562 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:22.672403 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-980b82-84d578bd5f-hz792" event={"ID":"50c5d765-70d6-467e-9400-0a79f379131c","Type":"ContainerStarted","Data":"32945a776bb31af1b15408062a530f934f8884f1d3e0f4dfdc0a26fdd5c77883"} Nov 02 23:33:22.673146 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:22.673113 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-980b82-84d578bd5f-hz792" Nov 02 23:33:22.788874 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:22.788801 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-980b82-84d578bd5f-hz792" podStartSLOduration=1.7887850090000001 podStartE2EDuration="1.788785009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:33:22.787539713 +0000 UTC m=+2689.026037401" watchObservedRunningTime="2026-11-02 23:33:22.788785009 +0000 UTC m=+2689.027282671" Nov 02 23:33:27.849034 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:33:27.848995 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:33:34.538501 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:34.538471 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 02 23:33:34.554103 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:34.554078 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 23:33:34.628056 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:34.628030 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 02 23:33:34.647460 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:34.647434 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 23:33:34.696351 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:34.694978 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-980b82-84d578bd5f-hz792" Nov 02 23:33:43.247652 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:43.247574 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz"] Nov 02 23:33:43.253314 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:43.253289 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz" Nov 02 23:33:43.311665 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:43.311636 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz"] Nov 02 23:33:43.319269 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:43.319242 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9j8d\" (UniqueName: \"kubernetes.io/projected/30badf9d-6a37-4a2a-ab5d-13c25bbec12c-kube-api-access-b9j8d\") pod \"maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz\" (UID: \"30badf9d-6a37-4a2a-ab5d-13c25bbec12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz" Nov 02 23:33:43.319414 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:43.319324 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30badf9d-6a37-4a2a-ab5d-13c25bbec12c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz\" (UID: \"30badf9d-6a37-4a2a-ab5d-13c25bbec12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz" Nov 02 23:33:43.319414 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:43.319385 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30badf9d-6a37-4a2a-ab5d-13c25bbec12c-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz\" (UID: \"30badf9d-6a37-4a2a-ab5d-13c25bbec12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz" Nov 02 23:33:43.319514 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:43.319494 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30badf9d-6a37-4a2a-ab5d-13c25bbec12c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz\" (UID: \"30badf9d-6a37-4a2a-ab5d-13c25bbec12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz" Nov 02 23:33:43.385176 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:43.385114 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-440c60-serving-cert\"" Nov 02 23:33:43.420649 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:43.420603 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9j8d\" (UniqueName: \"kubernetes.io/projected/30badf9d-6a37-4a2a-ab5d-13c25bbec12c-kube-api-access-b9j8d\") pod \"maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz\" (UID: \"30badf9d-6a37-4a2a-ab5d-13c25bbec12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz" Nov 02 23:33:43.420826 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:43.420691 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30badf9d-6a37-4a2a-ab5d-13c25bbec12c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz\" (UID: \"30badf9d-6a37-4a2a-ab5d-13c25bbec12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz" Nov 02 23:33:43.420826 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:43.420738 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30badf9d-6a37-4a2a-ab5d-13c25bbec12c-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz\" (UID: \"30badf9d-6a37-4a2a-ab5d-13c25bbec12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz" Nov 02 23:33:43.420826 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:43.420783 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30badf9d-6a37-4a2a-ab5d-13c25bbec12c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz\" (UID: \"30badf9d-6a37-4a2a-ab5d-13c25bbec12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz" Nov 02 23:33:43.421003 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:33:43.420913 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-440c60-serving-cert: secret "maas-api-e2e-shared-a-w0-440c60-serving-cert" not found Nov 02 23:33:43.421003 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:33:43.420982 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30badf9d-6a37-4a2a-ab5d-13c25bbec12c-maas-api-tls podName:30badf9d-6a37-4a2a-ab5d-13c25bbec12c nodeName:}" failed. No retries permitted until 2026-11-02 23:33:43.920959766 +0000 UTC m=+2710.159457419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/30badf9d-6a37-4a2a-ab5d-13c25bbec12c-maas-api-tls") pod "maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz" (UID: "30badf9d-6a37-4a2a-ab5d-13c25bbec12c") : secret "maas-api-e2e-shared-a-w0-440c60-serving-cert" not found Nov 02 23:33:43.421457 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:43.421429 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30badf9d-6a37-4a2a-ab5d-13c25bbec12c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz\" (UID: \"30badf9d-6a37-4a2a-ab5d-13c25bbec12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz" Nov 02 23:33:43.423275 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:43.423259 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30badf9d-6a37-4a2a-ab5d-13c25bbec12c-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz\" (UID: \"30badf9d-6a37-4a2a-ab5d-13c25bbec12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz" Nov 02 23:33:43.484002 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:43.483969 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9j8d\" (UniqueName: \"kubernetes.io/projected/30badf9d-6a37-4a2a-ab5d-13c25bbec12c-kube-api-access-b9j8d\") pod \"maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz\" (UID: \"30badf9d-6a37-4a2a-ab5d-13c25bbec12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz" Nov 02 23:33:43.926666 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:43.926634 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30badf9d-6a37-4a2a-ab5d-13c25bbec12c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz\" (UID: \"30badf9d-6a37-4a2a-ab5d-13c25bbec12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz" Nov 02 23:33:43.929497 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:43.929471 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30badf9d-6a37-4a2a-ab5d-13c25bbec12c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz\" (UID: \"30badf9d-6a37-4a2a-ab5d-13c25bbec12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz" Nov 02 23:33:44.167991 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:44.167954 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz" Nov 02 23:33:44.548305 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:44.548268 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz"] Nov 02 23:33:44.550031 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:33:44.550001 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30badf9d_6a37_4a2a_ab5d_13c25bbec12c.slice/crio-3b8bff805d66f5e8608b6ea5844e52ec234f22601ddcd4ed9b0a65568e7d3f7a WatchSource:0}: Error finding container 3b8bff805d66f5e8608b6ea5844e52ec234f22601ddcd4ed9b0a65568e7d3f7a: Status 404 returned error can't find the container with id 3b8bff805d66f5e8608b6ea5844e52ec234f22601ddcd4ed9b0a65568e7d3f7a Nov 02 23:33:44.551721 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:44.551698 5727 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 23:33:44.859777 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:44.859740 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz" event={"ID":"30badf9d-6a37-4a2a-ab5d-13c25bbec12c","Type":"ContainerStarted","Data":"3b8bff805d66f5e8608b6ea5844e52ec234f22601ddcd4ed9b0a65568e7d3f7a"} Nov 02 23:33:45.871286 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:45.871250 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz" event={"ID":"30badf9d-6a37-4a2a-ab5d-13c25bbec12c","Type":"ContainerStarted","Data":"fdb0f47bb8372ca5c0459cf5c89e4668eccb48b04482c7dded8c39a119050b8b"} Nov 02 23:33:45.871706 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:45.871478 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz" Nov 02 23:33:45.993301 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:45.993241 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz" podStartSLOduration=2.219385336 podStartE2EDuration="2.99322285s" totalImagesPullingTime="773.837514ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:33:45.98730067 +0000 UTC m=+2712.225798366" watchObservedRunningTime="2026-11-02 23:33:45.99322285 +0000 UTC m=+2712.231720512" Nov 02 23:33:51.893706 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:33:51.893677 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz" Nov 02 23:33:57.845841 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:33:57.845795 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:34:22.789316 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:22.789281 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv"] Nov 02 23:34:22.793866 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:22.793849 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv" Nov 02 23:34:22.839219 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:22.839180 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6df304dc-a952-436f-b0b4-8deda4e86884-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv\" (UID: \"6df304dc-a952-436f-b0b4-8deda4e86884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv" Nov 02 23:34:22.839359 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:22.839343 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6df304dc-a952-436f-b0b4-8deda4e86884-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv\" (UID: \"6df304dc-a952-436f-b0b4-8deda4e86884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv" Nov 02 23:34:22.839556 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:22.839532 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df304dc-a952-436f-b0b4-8deda4e86884-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv\" (UID: \"6df304dc-a952-436f-b0b4-8deda4e86884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv" Nov 02 23:34:22.839662 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:22.839644 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6nc2\" (UniqueName: \"kubernetes.io/projected/6df304dc-a952-436f-b0b4-8deda4e86884-kube-api-access-j6nc2\") pod \"maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv\" (UID: \"6df304dc-a952-436f-b0b4-8deda4e86884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv" Nov 02 23:34:22.846922 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:22.846898 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-980b82-serving-cert\"" Nov 02 23:34:22.900242 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:22.900214 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv"] Nov 02 23:34:22.941350 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:22.941317 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df304dc-a952-436f-b0b4-8deda4e86884-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv\" (UID: \"6df304dc-a952-436f-b0b4-8deda4e86884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv" Nov 02 23:34:22.941592 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:22.941560 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6nc2\" (UniqueName: \"kubernetes.io/projected/6df304dc-a952-436f-b0b4-8deda4e86884-kube-api-access-j6nc2\") pod \"maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv\" (UID: \"6df304dc-a952-436f-b0b4-8deda4e86884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv" Nov 02 23:34:22.941845 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:22.941822 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6df304dc-a952-436f-b0b4-8deda4e86884-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv\" (UID: \"6df304dc-a952-436f-b0b4-8deda4e86884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv" Nov 02 23:34:22.942092 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:22.942068 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6df304dc-a952-436f-b0b4-8deda4e86884-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv\" (UID: \"6df304dc-a952-436f-b0b4-8deda4e86884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv" Nov 02 23:34:22.942736 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:34:22.942170 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-980b82-serving-cert: secret "maas-api-e2e-shared-a-w2-980b82-serving-cert" not found Nov 02 23:34:22.942838 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:34:22.942788 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df304dc-a952-436f-b0b4-8deda4e86884-maas-api-tls podName:6df304dc-a952-436f-b0b4-8deda4e86884 nodeName:}" failed. No retries permitted until 2026-11-02 23:34:23.442766073 +0000 UTC m=+2749.681263718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6df304dc-a952-436f-b0b4-8deda4e86884-maas-api-tls") pod "maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv" (UID: "6df304dc-a952-436f-b0b4-8deda4e86884") : secret "maas-api-e2e-shared-a-w2-980b82-serving-cert" not found Nov 02 23:34:22.942838 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:22.942808 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6df304dc-a952-436f-b0b4-8deda4e86884-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv\" (UID: \"6df304dc-a952-436f-b0b4-8deda4e86884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv" Nov 02 23:34:22.953526 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:22.953500 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df304dc-a952-436f-b0b4-8deda4e86884-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv\" (UID: \"6df304dc-a952-436f-b0b4-8deda4e86884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv" Nov 02 23:34:23.005024 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:23.004995 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6nc2\" (UniqueName: \"kubernetes.io/projected/6df304dc-a952-436f-b0b4-8deda4e86884-kube-api-access-j6nc2\") pod \"maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv\" (UID: \"6df304dc-a952-436f-b0b4-8deda4e86884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv" Nov 02 23:34:23.447859 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:23.447824 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6df304dc-a952-436f-b0b4-8deda4e86884-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv\" (UID: \"6df304dc-a952-436f-b0b4-8deda4e86884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv" Nov 02 23:34:23.450519 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:23.450495 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6df304dc-a952-436f-b0b4-8deda4e86884-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv\" (UID: \"6df304dc-a952-436f-b0b4-8deda4e86884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv" Nov 02 23:34:23.708085 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:23.707993 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv" Nov 02 23:34:24.097370 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:34:24.097336 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df304dc_a952_436f_b0b4_8deda4e86884.slice/crio-e1f08eb386e34f1da4ceebdfe045974bb6ce7fc089918054014bc55534b30f69 WatchSource:0}: Error finding container e1f08eb386e34f1da4ceebdfe045974bb6ce7fc089918054014bc55534b30f69: Status 404 returned error can't find the container with id e1f08eb386e34f1da4ceebdfe045974bb6ce7fc089918054014bc55534b30f69 Nov 02 23:34:24.097781 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:24.097742 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv"] Nov 02 23:34:24.180532 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:24.180498 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv" event={"ID":"6df304dc-a952-436f-b0b4-8deda4e86884","Type":"ContainerStarted","Data":"e1f08eb386e34f1da4ceebdfe045974bb6ce7fc089918054014bc55534b30f69"} Nov 02 23:34:25.188609 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:25.188569 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv" event={"ID":"6df304dc-a952-436f-b0b4-8deda4e86884","Type":"ContainerStarted","Data":"199ef0c98da1dab4a61d7167808cfecb7e11e70bfc6ebbb84be7de4dadc4dc48"} Nov 02 23:34:25.189001 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:25.188639 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv" Nov 02 23:34:25.345745 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:25.345697 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv" podStartSLOduration=2.401495063 podStartE2EDuration="3.345682235s" totalImagesPullingTime="944.187172ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:34:25.343302095 +0000 UTC m=+2751.581799772" watchObservedRunningTime="2026-11-02 23:34:25.345682235 +0000 UTC m=+2751.584179897" Nov 02 23:34:27.846353 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:34:27.846310 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:34:31.202297 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:31.202263 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv" Nov 02 23:34:49.356509 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:49.356471 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a5d6d8-568d9db995-7v5c4"] Nov 02 23:34:49.361240 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:49.361218 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a5d6d8-568d9db995-7v5c4" Nov 02 23:34:49.406048 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:49.405989 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ec013db-b396-4d4e-8d00-5cf3f957b9c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a5d6d8-568d9db995-7v5c4\" (UID: \"4ec013db-b396-4d4e-8d00-5cf3f957b9c8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a5d6d8-568d9db995-7v5c4" Nov 02 23:34:49.407340 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:49.407312 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw486\" (UniqueName: \"kubernetes.io/projected/4ec013db-b396-4d4e-8d00-5cf3f957b9c8-kube-api-access-sw486\") pod \"payload-pre-processing-e2e-praxis-b-a5d6d8-568d9db995-7v5c4\" (UID: \"4ec013db-b396-4d4e-8d00-5cf3f957b9c8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a5d6d8-568d9db995-7v5c4" Nov 02 23:34:49.423098 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:49.423056 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a5d6d8-568d9db995-7v5c4"] Nov 02 23:34:49.462444 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:49.462412 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-a5d6d8\"" Nov 02 23:34:49.462444 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:49.462420 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-a5d6d8-dockercfg-cwmjz\"" Nov 02 23:34:49.508498 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:49.508455 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ec013db-b396-4d4e-8d00-5cf3f957b9c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a5d6d8-568d9db995-7v5c4\" (UID: \"4ec013db-b396-4d4e-8d00-5cf3f957b9c8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a5d6d8-568d9db995-7v5c4" Nov 02 23:34:49.508660 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:49.508637 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw486\" (UniqueName: \"kubernetes.io/projected/4ec013db-b396-4d4e-8d00-5cf3f957b9c8-kube-api-access-sw486\") pod \"payload-pre-processing-e2e-praxis-b-a5d6d8-568d9db995-7v5c4\" (UID: \"4ec013db-b396-4d4e-8d00-5cf3f957b9c8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a5d6d8-568d9db995-7v5c4" Nov 02 23:34:49.509139 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:49.509112 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ec013db-b396-4d4e-8d00-5cf3f957b9c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a5d6d8-568d9db995-7v5c4\" (UID: \"4ec013db-b396-4d4e-8d00-5cf3f957b9c8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a5d6d8-568d9db995-7v5c4" Nov 02 23:34:49.570230 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:49.570197 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw486\" (UniqueName: \"kubernetes.io/projected/4ec013db-b396-4d4e-8d00-5cf3f957b9c8-kube-api-access-sw486\") pod \"payload-pre-processing-e2e-praxis-b-a5d6d8-568d9db995-7v5c4\" (UID: \"4ec013db-b396-4d4e-8d00-5cf3f957b9c8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a5d6d8-568d9db995-7v5c4" Nov 02 23:34:49.683550 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:49.683463 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a5d6d8-568d9db995-7v5c4" Nov 02 23:34:50.066833 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:50.066803 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a5d6d8-568d9db995-7v5c4"] Nov 02 23:34:50.068651 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:34:50.068605 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ec013db_b396_4d4e_8d00_5cf3f957b9c8.slice/crio-52803860e19b08032228f729e64ceb003071c985cd78f5cc552f3d37f900017b WatchSource:0}: Error finding container 52803860e19b08032228f729e64ceb003071c985cd78f5cc552f3d37f900017b: Status 404 returned error can't find the container with id 52803860e19b08032228f729e64ceb003071c985cd78f5cc552f3d37f900017b Nov 02 23:34:50.393603 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:50.393508 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a5d6d8-568d9db995-7v5c4" event={"ID":"4ec013db-b396-4d4e-8d00-5cf3f957b9c8","Type":"ContainerStarted","Data":"f5aa4d27515214562f629fe968a46f9c84e6b064c8c9fe7f3ef778ad183cdf74"} Nov 02 23:34:50.393603 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:50.393565 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a5d6d8-568d9db995-7v5c4" event={"ID":"4ec013db-b396-4d4e-8d00-5cf3f957b9c8","Type":"ContainerStarted","Data":"52803860e19b08032228f729e64ceb003071c985cd78f5cc552f3d37f900017b"} Nov 02 23:34:50.394046 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:50.393627 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a5d6d8-568d9db995-7v5c4" Nov 02 23:34:50.511306 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:34:50.511256 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a5d6d8-568d9db995-7v5c4" podStartSLOduration=1.51124159 podStartE2EDuration="1.51124159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:34:50.509214355 +0000 UTC m=+2776.747712041" watchObservedRunningTime="2026-11-02 23:34:50.51124159 +0000 UTC m=+2776.749739252" Nov 02 23:34:57.846655 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:34:57.846620 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:35:01.481118 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:01.481089 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a5d6d8-568d9db995-7v5c4" Nov 02 23:35:06.561539 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:06.561502 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-915ccb-57c5454f5c-ggx2n"] Nov 02 23:35:06.568392 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:06.568119 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-915ccb-57c5454f5c-ggx2n" Nov 02 23:35:06.620182 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:06.620147 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-915ccb-57c5454f5c-ggx2n"] Nov 02 23:35:06.665393 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:06.665359 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-915ccb\"" Nov 02 23:35:06.665593 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:06.665482 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-915ccb-dockercfg-f2qd8\"" Nov 02 23:35:06.687289 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:06.687263 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56crt\" (UniqueName: \"kubernetes.io/projected/0a89e95d-54df-4506-94ec-34f333911ef7-kube-api-access-56crt\") pod \"payload-processing-e2e-shared-b-w0-915ccb-57c5454f5c-ggx2n\" (UID: \"0a89e95d-54df-4506-94ec-34f333911ef7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-915ccb-57c5454f5c-ggx2n" Nov 02 23:35:06.687442 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:06.687306 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a89e95d-54df-4506-94ec-34f333911ef7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-915ccb-57c5454f5c-ggx2n\" (UID: \"0a89e95d-54df-4506-94ec-34f333911ef7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-915ccb-57c5454f5c-ggx2n" Nov 02 23:35:06.788259 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:06.788225 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56crt\" (UniqueName: \"kubernetes.io/projected/0a89e95d-54df-4506-94ec-34f333911ef7-kube-api-access-56crt\") pod \"payload-processing-e2e-shared-b-w0-915ccb-57c5454f5c-ggx2n\" (UID: \"0a89e95d-54df-4506-94ec-34f333911ef7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-915ccb-57c5454f5c-ggx2n" Nov 02 23:35:06.788460 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:06.788276 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a89e95d-54df-4506-94ec-34f333911ef7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-915ccb-57c5454f5c-ggx2n\" (UID: \"0a89e95d-54df-4506-94ec-34f333911ef7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-915ccb-57c5454f5c-ggx2n" Nov 02 23:35:06.789015 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:06.788993 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a89e95d-54df-4506-94ec-34f333911ef7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-915ccb-57c5454f5c-ggx2n\" (UID: \"0a89e95d-54df-4506-94ec-34f333911ef7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-915ccb-57c5454f5c-ggx2n" Nov 02 23:35:06.852536 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:06.852496 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56crt\" (UniqueName: \"kubernetes.io/projected/0a89e95d-54df-4506-94ec-34f333911ef7-kube-api-access-56crt\") pod \"payload-processing-e2e-shared-b-w0-915ccb-57c5454f5c-ggx2n\" (UID: \"0a89e95d-54df-4506-94ec-34f333911ef7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-915ccb-57c5454f5c-ggx2n" Nov 02 23:35:06.889865 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:06.889834 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-915ccb-57c5454f5c-ggx2n" Nov 02 23:35:07.278900 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:07.278861 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-915ccb-57c5454f5c-ggx2n"] Nov 02 23:35:07.279343 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:35:07.279301 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a89e95d_54df_4506_94ec_34f333911ef7.slice/crio-2d3abd952ca1a79e81aa3a859889c3d10c44a098a1f772c33c9874d36d40bcf3 WatchSource:0}: Error finding container 2d3abd952ca1a79e81aa3a859889c3d10c44a098a1f772c33c9874d36d40bcf3: Status 404 returned error can't find the container with id 2d3abd952ca1a79e81aa3a859889c3d10c44a098a1f772c33c9874d36d40bcf3 Nov 02 23:35:07.605807 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:07.605773 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-915ccb-57c5454f5c-ggx2n" event={"ID":"0a89e95d-54df-4506-94ec-34f333911ef7","Type":"ContainerStarted","Data":"f3bc723ae3ee5c85cd2b81500071bc2c904ca453f4e15ac7412b064018301ca3"} Nov 02 23:35:07.606227 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:07.605830 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-915ccb-57c5454f5c-ggx2n" event={"ID":"0a89e95d-54df-4506-94ec-34f333911ef7","Type":"ContainerStarted","Data":"2d3abd952ca1a79e81aa3a859889c3d10c44a098a1f772c33c9874d36d40bcf3"} Nov 02 23:35:07.606227 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:07.605980 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-915ccb-57c5454f5c-ggx2n" Nov 02 23:35:07.724909 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:07.724863 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-915ccb-57c5454f5c-ggx2n" podStartSLOduration=1.724848894 podStartE2EDuration="1.724848894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:35:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:35:07.721103274 +0000 UTC m=+2793.959600967" watchObservedRunningTime="2026-11-02 23:35:07.724848894 +0000 UTC m=+2793.963346556" Nov 02 23:35:16.050039 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:16.049953 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c84dbcbd-w72p6"] Nov 02 23:35:16.100576 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:16.100520 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c84dbcbd-w72p6"] Nov 02 23:35:17.884107 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:17.884076 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="843c2835-6862-44c8-ac4e-2d5cd3806fa8" path="/var/lib/kubelet/pods/843c2835-6862-44c8-ac4e-2d5cd3806fa8/volumes" Nov 02 23:35:18.615690 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:18.615658 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-915ccb-57c5454f5c-ggx2n" Nov 02 23:35:27.846376 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:35:27.846334 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:35:29.447878 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:29.447839 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj"] Nov 02 23:35:29.448397 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:29.448082 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" podUID="c5c0fb24-5f82-4c08-bc9b-f80476f4ded9" containerName="istio-proxy" containerID="cri-o://f15ea955500c4e2237ee2c82e3d19c7074202d51608e7928de28df1ba4ff0cd5" gracePeriod=30 Nov 02 23:35:29.746997 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:29.746916 5727 patch_prober.go:29] interesting pod/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:35:29.746997 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:29.746965 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" podUID="c5c0fb24-5f82-4c08-bc9b-f80476f4ded9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:35:34.730547 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.730487 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:35:34.814443 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.814400 5727 generic.go:356] "Generic (PLEG): container finished" podID="c5c0fb24-5f82-4c08-bc9b-f80476f4ded9" containerID="f15ea955500c4e2237ee2c82e3d19c7074202d51608e7928de28df1ba4ff0cd5" exitCode=0 Nov 02 23:35:34.814618 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.814454 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" Nov 02 23:35:34.814618 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.814466 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" event={"ID":"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9","Type":"ContainerDied","Data":"f15ea955500c4e2237ee2c82e3d19c7074202d51608e7928de28df1ba4ff0cd5"} Nov 02 23:35:34.814618 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.814522 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj" event={"ID":"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9","Type":"ContainerDied","Data":"e470cbb49c5f76e9063fa5ff41177c4a8632c7f87417a0b72b234fc414667d92"} Nov 02 23:35:34.814618 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.814551 5727 scope.go:122] "RemoveContainer" containerID="f15ea955500c4e2237ee2c82e3d19c7074202d51608e7928de28df1ba4ff0cd5" Nov 02 23:35:34.828581 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.828559 5727 scope.go:122] "RemoveContainer" containerID="f15ea955500c4e2237ee2c82e3d19c7074202d51608e7928de28df1ba4ff0cd5" Nov 02 23:35:34.829036 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:35:34.829012 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f15ea955500c4e2237ee2c82e3d19c7074202d51608e7928de28df1ba4ff0cd5\": container with ID starting with f15ea955500c4e2237ee2c82e3d19c7074202d51608e7928de28df1ba4ff0cd5 not found: ID does not exist" containerID="f15ea955500c4e2237ee2c82e3d19c7074202d51608e7928de28df1ba4ff0cd5" Nov 02 23:35:34.829170 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.829044 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f15ea955500c4e2237ee2c82e3d19c7074202d51608e7928de28df1ba4ff0cd5"} err="failed to get container status \"f15ea955500c4e2237ee2c82e3d19c7074202d51608e7928de28df1ba4ff0cd5\": rpc error: code = NotFound desc = could not find container \"f15ea955500c4e2237ee2c82e3d19c7074202d51608e7928de28df1ba4ff0cd5\": container with ID starting with f15ea955500c4e2237ee2c82e3d19c7074202d51608e7928de28df1ba4ff0cd5 not found: ID does not exist" Nov 02 23:35:34.872583 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.872472 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istiod-ca-cert\") pod \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " Nov 02 23:35:34.872583 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.872526 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-credential-socket\") pod \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " Nov 02 23:35:34.872731 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.872606 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-workload-socket\") pod \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " Nov 02 23:35:34.873268 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.873021 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istio-token\" (UniqueName: \"kubernetes.io/projected/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istio-token\") pod \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " Nov 02 23:35:34.873268 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.873076 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istio-data\") pod \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " Nov 02 23:35:34.873268 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.873161 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-workload-certs\") pod \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " Nov 02 23:35:34.873268 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.873182 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-credential-socket" pod "c5c0fb24-5f82-4c08-bc9b-f80476f4ded9" (UID: "c5c0fb24-5f82-4c08-bc9b-f80476f4ded9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:34.873268 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.873218 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-workload-socket" pod "c5c0fb24-5f82-4c08-bc9b-f80476f4ded9" (UID: "c5c0fb24-5f82-4c08-bc9b-f80476f4ded9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:34.873268 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.873168 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istiod-ca-cert" pod "c5c0fb24-5f82-4c08-bc9b-f80476f4ded9" (UID: "c5c0fb24-5f82-4c08-bc9b-f80476f4ded9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:35:34.873632 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.873312 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istio-podinfo\") pod \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " Nov 02 23:35:34.873632 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.873422 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-kube-api-access-6dxj7\" (UniqueName: \"kubernetes.io/projected/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-kube-api-access-6dxj7\") pod \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " Nov 02 23:35:34.873632 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.873421 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-workload-certs" pod "c5c0fb24-5f82-4c08-bc9b-f80476f4ded9" (UID: "c5c0fb24-5f82-4c08-bc9b-f80476f4ded9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:34.873632 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.873435 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istio-data" pod "c5c0fb24-5f82-4c08-bc9b-f80476f4ded9" (UID: "c5c0fb24-5f82-4c08-bc9b-f80476f4ded9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:34.873632 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.873460 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istio-envoy\") pod \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\" (UID: \"c5c0fb24-5f82-4c08-bc9b-f80476f4ded9\") " Nov 02 23:35:34.874226 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.874206 5727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istiod-ca-cert\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:34.874319 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.874231 5727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-credential-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:34.874319 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.874246 5727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-workload-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:34.874319 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.874259 5727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istio-data\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:34.874319 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.874278 5727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-workload-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:34.877406 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.877382 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istio-podinfo" pod "c5c0fb24-5f82-4c08-bc9b-f80476f4ded9" (UID: "c5c0fb24-5f82-4c08-bc9b-f80476f4ded9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:35:34.878290 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.878263 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istio-token" pod "c5c0fb24-5f82-4c08-bc9b-f80476f4ded9" (UID: "c5c0fb24-5f82-4c08-bc9b-f80476f4ded9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:35:34.879259 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.879237 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istio-envoy" pod "c5c0fb24-5f82-4c08-bc9b-f80476f4ded9" (UID: "c5c0fb24-5f82-4c08-bc9b-f80476f4ded9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:34.879348 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.879320 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-kube-api-access-6dxj7" pod "c5c0fb24-5f82-4c08-bc9b-f80476f4ded9" (UID: "c5c0fb24-5f82-4c08-bc9b-f80476f4ded9"). InnerVolumeSpecName "kube-api-access-6dxj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:35:34.975576 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.975541 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dxj7\" (UniqueName: \"kubernetes.io/projected/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-kube-api-access-6dxj7\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:34.975576 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.975577 5727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istio-envoy\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:34.975822 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.975595 5727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istio-token\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:34.975822 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:34.975610 5727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9-istio-podinfo\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:35.237250 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:35.237203 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj"] Nov 02 23:35:35.242042 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:35.242013 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-kbcvj"] Nov 02 23:35:35.884946 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:35.884905 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5c0fb24-5f82-4c08-bc9b-f80476f4ded9" path="/var/lib/kubelet/pods/c5c0fb24-5f82-4c08-bc9b-f80476f4ded9/volumes" Nov 02 23:35:45.020797 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:45.020762 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8d3cb-679d85bcf-5mkkh"] Nov 02 23:35:45.021481 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:45.021465 5727 cpu_manager.go:414] "containerMap: removing container" podUID="c5c0fb24-5f82-4c08-bc9b-f80476f4ded9" containerName="istio-proxy" Nov 02 23:35:45.021529 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:45.021483 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5c0fb24-5f82-4c08-bc9b-f80476f4ded9" containerName="istio-proxy" Nov 02 23:35:45.021623 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:45.021614 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5c0fb24-5f82-4c08-bc9b-f80476f4ded9" containerName="istio-proxy" Nov 02 23:35:45.026681 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:45.026662 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8d3cb-679d85bcf-5mkkh" Nov 02 23:35:45.082734 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:45.082702 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8d3cb-679d85bcf-5mkkh"] Nov 02 23:35:45.111628 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:45.111595 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fbzv\" (UniqueName: \"kubernetes.io/projected/6206423f-eac2-4922-89fe-f5632e0fe671-kube-api-access-6fbzv\") pod \"payload-pre-processing-e2e-shared-b-w2-b8d3cb-679d85bcf-5mkkh\" (UID: \"6206423f-eac2-4922-89fe-f5632e0fe671\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8d3cb-679d85bcf-5mkkh" Nov 02 23:35:45.111783 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:45.111730 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6206423f-eac2-4922-89fe-f5632e0fe671-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b8d3cb-679d85bcf-5mkkh\" (UID: \"6206423f-eac2-4922-89fe-f5632e0fe671\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8d3cb-679d85bcf-5mkkh" Nov 02 23:35:45.127311 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:45.127275 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-b8d3cb\"" Nov 02 23:35:45.127456 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:45.127378 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-b8d3cb-dockercfg-72bqw\"" Nov 02 23:35:45.214010 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:45.213975 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fbzv\" (UniqueName: \"kubernetes.io/projected/6206423f-eac2-4922-89fe-f5632e0fe671-kube-api-access-6fbzv\") pod \"payload-pre-processing-e2e-shared-b-w2-b8d3cb-679d85bcf-5mkkh\" (UID: \"6206423f-eac2-4922-89fe-f5632e0fe671\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8d3cb-679d85bcf-5mkkh" Nov 02 23:35:45.214678 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:45.214647 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6206423f-eac2-4922-89fe-f5632e0fe671-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b8d3cb-679d85bcf-5mkkh\" (UID: \"6206423f-eac2-4922-89fe-f5632e0fe671\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8d3cb-679d85bcf-5mkkh" Nov 02 23:35:45.217451 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:45.216685 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6206423f-eac2-4922-89fe-f5632e0fe671-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b8d3cb-679d85bcf-5mkkh\" (UID: \"6206423f-eac2-4922-89fe-f5632e0fe671\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8d3cb-679d85bcf-5mkkh" Nov 02 23:35:45.286628 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:45.286557 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fbzv\" (UniqueName: \"kubernetes.io/projected/6206423f-eac2-4922-89fe-f5632e0fe671-kube-api-access-6fbzv\") pod \"payload-pre-processing-e2e-shared-b-w2-b8d3cb-679d85bcf-5mkkh\" (UID: \"6206423f-eac2-4922-89fe-f5632e0fe671\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8d3cb-679d85bcf-5mkkh" Nov 02 23:35:45.341289 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:45.341256 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8d3cb-679d85bcf-5mkkh" Nov 02 23:35:45.726790 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:45.726762 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8d3cb-679d85bcf-5mkkh"] Nov 02 23:35:45.727458 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:35:45.727425 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6206423f_eac2_4922_89fe_f5632e0fe671.slice/crio-0cf7ed1f68797fc9e8e26912cd635042ea0ddf97d8f9ff99c925f449ee3861eb WatchSource:0}: Error finding container 0cf7ed1f68797fc9e8e26912cd635042ea0ddf97d8f9ff99c925f449ee3861eb: Status 404 returned error can't find the container with id 0cf7ed1f68797fc9e8e26912cd635042ea0ddf97d8f9ff99c925f449ee3861eb Nov 02 23:35:45.913295 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:45.913256 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8d3cb-679d85bcf-5mkkh" event={"ID":"6206423f-eac2-4922-89fe-f5632e0fe671","Type":"ContainerStarted","Data":"3c3615939cca7c765ae7a7ae1f7ad2aa9e1eb3fba871d49d95c709c5cf518055"} Nov 02 23:35:45.913448 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:45.913316 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8d3cb-679d85bcf-5mkkh" event={"ID":"6206423f-eac2-4922-89fe-f5632e0fe671","Type":"ContainerStarted","Data":"0cf7ed1f68797fc9e8e26912cd635042ea0ddf97d8f9ff99c925f449ee3861eb"} Nov 02 23:35:45.913448 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:45.913360 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8d3cb-679d85bcf-5mkkh" Nov 02 23:35:46.028951 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:46.028850 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8d3cb-679d85bcf-5mkkh" podStartSLOduration=2.028836784 podStartE2EDuration="2.028836784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:35:46.027587787 +0000 UTC m=+2832.266085452" watchObservedRunningTime="2026-11-02 23:35:46.028836784 +0000 UTC m=+2832.267334473" Nov 02 23:35:56.926548 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:35:56.926521 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8d3cb-679d85bcf-5mkkh" Nov 02 23:35:57.846891 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:35:57.846855 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:36:08.519195 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:08.519162 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp"] Nov 02 23:36:08.523598 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:08.523580 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp" Nov 02 23:36:08.575850 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:08.575812 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-915ccb-serving-cert\"" Nov 02 23:36:08.616807 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:08.616775 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21dd429a-84f5-4a14-b485-51aed38a3748-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp\" (UID: \"21dd429a-84f5-4a14-b485-51aed38a3748\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp" Nov 02 23:36:08.617001 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:08.616899 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x8hn\" (UniqueName: \"kubernetes.io/projected/21dd429a-84f5-4a14-b485-51aed38a3748-kube-api-access-9x8hn\") pod \"maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp\" (UID: \"21dd429a-84f5-4a14-b485-51aed38a3748\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp" Nov 02 23:36:08.617076 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:08.617046 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21dd429a-84f5-4a14-b485-51aed38a3748-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp\" (UID: \"21dd429a-84f5-4a14-b485-51aed38a3748\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp" Nov 02 23:36:08.617170 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:08.617095 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21dd429a-84f5-4a14-b485-51aed38a3748-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp\" (UID: \"21dd429a-84f5-4a14-b485-51aed38a3748\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp" Nov 02 23:36:08.629108 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:08.629082 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp"] Nov 02 23:36:08.718662 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:08.718631 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21dd429a-84f5-4a14-b485-51aed38a3748-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp\" (UID: \"21dd429a-84f5-4a14-b485-51aed38a3748\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp" Nov 02 23:36:08.718850 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:08.718729 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x8hn\" (UniqueName: \"kubernetes.io/projected/21dd429a-84f5-4a14-b485-51aed38a3748-kube-api-access-9x8hn\") pod \"maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp\" (UID: \"21dd429a-84f5-4a14-b485-51aed38a3748\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp" Nov 02 23:36:08.718850 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:08.718804 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21dd429a-84f5-4a14-b485-51aed38a3748-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp\" (UID: \"21dd429a-84f5-4a14-b485-51aed38a3748\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp" Nov 02 23:36:08.718850 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:36:08.718805 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-915ccb-serving-cert: secret "maas-api-e2e-shared-b-w0-915ccb-serving-cert" not found Nov 02 23:36:08.718850 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:08.718844 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21dd429a-84f5-4a14-b485-51aed38a3748-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp\" (UID: \"21dd429a-84f5-4a14-b485-51aed38a3748\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp" Nov 02 23:36:08.719177 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:36:08.718896 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21dd429a-84f5-4a14-b485-51aed38a3748-maas-api-tls podName:21dd429a-84f5-4a14-b485-51aed38a3748 nodeName:}" failed. No retries permitted until 2026-11-02 23:36:09.218873978 +0000 UTC m=+2855.457371619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/21dd429a-84f5-4a14-b485-51aed38a3748-maas-api-tls") pod "maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp" (UID: "21dd429a-84f5-4a14-b485-51aed38a3748") : secret "maas-api-e2e-shared-b-w0-915ccb-serving-cert" not found Nov 02 23:36:08.720004 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:08.719977 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21dd429a-84f5-4a14-b485-51aed38a3748-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp\" (UID: \"21dd429a-84f5-4a14-b485-51aed38a3748\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp" Nov 02 23:36:08.721770 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:08.721740 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21dd429a-84f5-4a14-b485-51aed38a3748-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp\" (UID: \"21dd429a-84f5-4a14-b485-51aed38a3748\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp" Nov 02 23:36:08.781704 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:08.781624 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x8hn\" (UniqueName: \"kubernetes.io/projected/21dd429a-84f5-4a14-b485-51aed38a3748-kube-api-access-9x8hn\") pod \"maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp\" (UID: \"21dd429a-84f5-4a14-b485-51aed38a3748\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp" Nov 02 23:36:09.226294 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:09.226262 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21dd429a-84f5-4a14-b485-51aed38a3748-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp\" (UID: \"21dd429a-84f5-4a14-b485-51aed38a3748\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp" Nov 02 23:36:09.229064 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:09.229036 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21dd429a-84f5-4a14-b485-51aed38a3748-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp\" (UID: \"21dd429a-84f5-4a14-b485-51aed38a3748\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp" Nov 02 23:36:09.437459 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:09.437412 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp" Nov 02 23:36:09.836566 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:09.836536 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp"] Nov 02 23:36:09.837736 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:36:09.837709 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21dd429a_84f5_4a14_b485_51aed38a3748.slice/crio-9a231b4b5806ac400fb593b90ef82707ebe717f719681921aa6b167f79572eeb WatchSource:0}: Error finding container 9a231b4b5806ac400fb593b90ef82707ebe717f719681921aa6b167f79572eeb: Status 404 returned error can't find the container with id 9a231b4b5806ac400fb593b90ef82707ebe717f719681921aa6b167f79572eeb Nov 02 23:36:10.144043 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:10.143960 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp" event={"ID":"21dd429a-84f5-4a14-b485-51aed38a3748","Type":"ContainerStarted","Data":"9a231b4b5806ac400fb593b90ef82707ebe717f719681921aa6b167f79572eeb"} Nov 02 23:36:11.164052 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:11.162215 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp" event={"ID":"21dd429a-84f5-4a14-b485-51aed38a3748","Type":"ContainerStarted","Data":"c3cfd3c23de6be2862bf9e36cd5e971d0cab8190da13cbeb67dbcfc29512dc93"} Nov 02 23:36:11.164052 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:11.163411 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp" Nov 02 23:36:11.280050 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:11.279975 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp" podStartSLOduration=2.4324951280000002 podStartE2EDuration="3.279956574s" totalImagesPullingTime="847.461446ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:36:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:36:11.278935625 +0000 UTC m=+2857.517433306" watchObservedRunningTime="2026-11-02 23:36:11.279956574 +0000 UTC m=+2857.518454238" Nov 02 23:36:18.184735 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:18.184706 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp" Nov 02 23:36:19.199390 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:19.199356 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4cdec0-65f74dfb55-6896h"] Nov 02 23:36:19.204936 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:19.204914 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cdec0-65f74dfb55-6896h" Nov 02 23:36:19.259758 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:19.259721 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4cdec0-65f74dfb55-6896h"] Nov 02 23:36:19.268606 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:19.268581 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqjkn\" (UniqueName: \"kubernetes.io/projected/7abe7663-f846-4499-b6d4-0c85ab2a148d-kube-api-access-mqjkn\") pod \"payload-pre-processing-e2e-praxis-a-4cdec0-65f74dfb55-6896h\" (UID: \"7abe7663-f846-4499-b6d4-0c85ab2a148d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cdec0-65f74dfb55-6896h" Nov 02 23:36:19.268946 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:19.268917 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7abe7663-f846-4499-b6d4-0c85ab2a148d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4cdec0-65f74dfb55-6896h\" (UID: \"7abe7663-f846-4499-b6d4-0c85ab2a148d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cdec0-65f74dfb55-6896h" Nov 02 23:36:19.321887 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:19.319503 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-4cdec0-dockercfg-fdsjz\"" Nov 02 23:36:19.321887 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:19.320246 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-4cdec0\"" Nov 02 23:36:19.361892 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:19.361834 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4cdec0-ccc48bdc7-dkvtg"] Nov 02 23:36:19.376980 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:19.375182 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqjkn\" (UniqueName: \"kubernetes.io/projected/7abe7663-f846-4499-b6d4-0c85ab2a148d-kube-api-access-mqjkn\") pod \"payload-pre-processing-e2e-praxis-a-4cdec0-65f74dfb55-6896h\" (UID: \"7abe7663-f846-4499-b6d4-0c85ab2a148d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cdec0-65f74dfb55-6896h" Nov 02 23:36:19.376980 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:19.375390 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7abe7663-f846-4499-b6d4-0c85ab2a148d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4cdec0-65f74dfb55-6896h\" (UID: \"7abe7663-f846-4499-b6d4-0c85ab2a148d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cdec0-65f74dfb55-6896h" Nov 02 23:36:19.376980 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:19.376396 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7abe7663-f846-4499-b6d4-0c85ab2a148d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4cdec0-65f74dfb55-6896h\" (UID: \"7abe7663-f846-4499-b6d4-0c85ab2a148d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cdec0-65f74dfb55-6896h" Nov 02 23:36:19.396149 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:19.393491 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cdec0-ccc48bdc7-dkvtg" Nov 02 23:36:19.436055 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:19.436024 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4cdec0-ccc48bdc7-dkvtg"] Nov 02 23:36:19.439452 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:19.439425 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqjkn\" (UniqueName: \"kubernetes.io/projected/7abe7663-f846-4499-b6d4-0c85ab2a148d-kube-api-access-mqjkn\") pod \"payload-pre-processing-e2e-praxis-a-4cdec0-65f74dfb55-6896h\" (UID: \"7abe7663-f846-4499-b6d4-0c85ab2a148d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cdec0-65f74dfb55-6896h" Nov 02 23:36:19.476710 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:19.476616 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgggw\" (UniqueName: \"kubernetes.io/projected/f0b5de66-8158-4fb5-b340-d7a5b85f2766-kube-api-access-tgggw\") pod \"payload-processing-e2e-praxis-a-4cdec0-ccc48bdc7-dkvtg\" (UID: \"f0b5de66-8158-4fb5-b340-d7a5b85f2766\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4cdec0-ccc48bdc7-dkvtg" Nov 02 23:36:19.476941 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:19.476913 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b5de66-8158-4fb5-b340-d7a5b85f2766-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4cdec0-ccc48bdc7-dkvtg\" (UID: \"f0b5de66-8158-4fb5-b340-d7a5b85f2766\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4cdec0-ccc48bdc7-dkvtg" Nov 02 23:36:19.520372 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:19.520334 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cdec0-65f74dfb55-6896h" Nov 02 23:36:19.579763 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:19.578950 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b5de66-8158-4fb5-b340-d7a5b85f2766-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4cdec0-ccc48bdc7-dkvtg\" (UID: \"f0b5de66-8158-4fb5-b340-d7a5b85f2766\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4cdec0-ccc48bdc7-dkvtg" Nov 02 23:36:19.579763 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:19.579083 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgggw\" (UniqueName: \"kubernetes.io/projected/f0b5de66-8158-4fb5-b340-d7a5b85f2766-kube-api-access-tgggw\") pod \"payload-processing-e2e-praxis-a-4cdec0-ccc48bdc7-dkvtg\" (UID: \"f0b5de66-8158-4fb5-b340-d7a5b85f2766\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4cdec0-ccc48bdc7-dkvtg" Nov 02 23:36:19.580696 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:19.580664 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b5de66-8158-4fb5-b340-d7a5b85f2766-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4cdec0-ccc48bdc7-dkvtg\" (UID: \"f0b5de66-8158-4fb5-b340-d7a5b85f2766\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4cdec0-ccc48bdc7-dkvtg" Nov 02 23:36:19.640713 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:19.640676 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgggw\" (UniqueName: \"kubernetes.io/projected/f0b5de66-8158-4fb5-b340-d7a5b85f2766-kube-api-access-tgggw\") pod \"payload-processing-e2e-praxis-a-4cdec0-ccc48bdc7-dkvtg\" (UID: \"f0b5de66-8158-4fb5-b340-d7a5b85f2766\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4cdec0-ccc48bdc7-dkvtg" Nov 02 23:36:19.714108 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:19.714073 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cdec0-ccc48bdc7-dkvtg" Nov 02 23:36:19.907377 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:19.907344 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4cdec0-65f74dfb55-6896h"] Nov 02 23:36:19.908421 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:36:19.908391 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7abe7663_f846_4499_b6d4_0c85ab2a148d.slice/crio-1b0f9da6cbd564cc2261b494f4a8454fbbafcb2eb3d29444576931902dfacc41 WatchSource:0}: Error finding container 1b0f9da6cbd564cc2261b494f4a8454fbbafcb2eb3d29444576931902dfacc41: Status 404 returned error can't find the container with id 1b0f9da6cbd564cc2261b494f4a8454fbbafcb2eb3d29444576931902dfacc41 Nov 02 23:36:20.125741 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:20.125713 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4cdec0-ccc48bdc7-dkvtg"] Nov 02 23:36:20.126290 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:36:20.126268 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b5de66_8158_4fb5_b340_d7a5b85f2766.slice/crio-09735015aa76c39a76abdea3449db8cce0ea4633d3da1d8de83dfac07fbd24b6 WatchSource:0}: Error finding container 09735015aa76c39a76abdea3449db8cce0ea4633d3da1d8de83dfac07fbd24b6: Status 404 returned error can't find the container with id 09735015aa76c39a76abdea3449db8cce0ea4633d3da1d8de83dfac07fbd24b6 Nov 02 23:36:20.234120 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:20.234084 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cdec0-65f74dfb55-6896h" event={"ID":"7abe7663-f846-4499-b6d4-0c85ab2a148d","Type":"ContainerStarted","Data":"828b080ad38ee8e135f44af3bb3e58e4f44afea6f622c7d6b0611f0aaca804f9"} Nov 02 23:36:20.234537 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:20.234158 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cdec0-65f74dfb55-6896h" event={"ID":"7abe7663-f846-4499-b6d4-0c85ab2a148d","Type":"ContainerStarted","Data":"1b0f9da6cbd564cc2261b494f4a8454fbbafcb2eb3d29444576931902dfacc41"} Nov 02 23:36:20.234537 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:20.234208 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cdec0-65f74dfb55-6896h" Nov 02 23:36:20.235917 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:20.235893 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cdec0-ccc48bdc7-dkvtg" event={"ID":"f0b5de66-8158-4fb5-b340-d7a5b85f2766","Type":"ContainerStarted","Data":"d13dd821c2a4626b5ba67c049704e1fc36804e3c3841bb4ceb2167df350ff3ac"} Nov 02 23:36:20.236024 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:20.235929 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cdec0-ccc48bdc7-dkvtg" event={"ID":"f0b5de66-8158-4fb5-b340-d7a5b85f2766","Type":"ContainerStarted","Data":"09735015aa76c39a76abdea3449db8cce0ea4633d3da1d8de83dfac07fbd24b6"} Nov 02 23:36:20.236024 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:20.235949 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cdec0-ccc48bdc7-dkvtg" Nov 02 23:36:20.349716 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:20.349667 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cdec0-65f74dfb55-6896h" podStartSLOduration=1.349654023 podStartE2EDuration="1.349654023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:36:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:36:20.348216588 +0000 UTC m=+2866.586714249" watchObservedRunningTime="2026-11-02 23:36:20.349654023 +0000 UTC m=+2866.588151685" Nov 02 23:36:20.468071 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:20.467952 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cdec0-ccc48bdc7-dkvtg" podStartSLOduration=1.467934576 podStartE2EDuration="1.467934576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:36:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:36:20.46201893 +0000 UTC m=+2866.700516606" watchObservedRunningTime="2026-11-02 23:36:20.467934576 +0000 UTC m=+2866.706432238" Nov 02 23:36:27.848726 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:36:27.848687 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:36:31.247413 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:31.247383 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cdec0-ccc48bdc7-dkvtg" Nov 02 23:36:31.247885 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:31.247454 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cdec0-65f74dfb55-6896h" Nov 02 23:36:34.882567 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:36:34.882529 5727 scope.go:122] "RemoveContainer" containerID="62215ffc337101c9d54db54c8b3b900c0686d18c7029fa64bd772b6154c44720" Nov 02 23:36:57.847038 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:36:57.846998 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:37:27.847177 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:37:27.847109 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:37:57.846938 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:37:57.846898 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:38:27.846601 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:38:27.846562 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:38:29.715002 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:29.714965 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2"] Nov 02 23:38:29.715471 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:29.715218 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" podUID="cabff31e-d3dd-469a-8f1d-f4e8fd22fe18" containerName="istio-proxy" containerID="cri-o://470c9d03cb4a9a50f315a394a7b662107a6ad1cda9b858eb14328283fb826a7b" gracePeriod=30 Nov 02 23:38:32.367105 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:32.367067 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z"] Nov 02 23:38:32.371739 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:32.371722 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" Nov 02 23:38:32.439171 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:32.439082 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-440c60\"/\"openshift-service-ca.crt\"" Nov 02 23:38:32.445238 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:32.445204 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z"] Nov 02 23:38:32.474314 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:32.474274 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-440c60\"/\"default-dockercfg-22f76\"" Nov 02 23:38:32.474314 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:32.474280 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-440c60\"/\"kube-root-ca.crt\"" Nov 02 23:38:32.474558 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:32.474442 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-440c60\"/\"auth-test-model-440c60-kserve-self-signed-certs\"" Nov 02 23:38:32.524947 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:32.524905 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fcc517d-e223-4597-99c4-9deaaf163659-dshm\") pod \"auth-test-model-440c60-kserve-5579dbbdbf-2mh9z\" (UID: \"0fcc517d-e223-4597-99c4-9deaaf163659\") " pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" Nov 02 23:38:32.525227 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:32.525053 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppfmk\" (UniqueName: \"kubernetes.io/projected/0fcc517d-e223-4597-99c4-9deaaf163659-kube-api-access-ppfmk\") pod \"auth-test-model-440c60-kserve-5579dbbdbf-2mh9z\" (UID: \"0fcc517d-e223-4597-99c4-9deaaf163659\") " pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" Nov 02 23:38:32.525298 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:32.525251 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fcc517d-e223-4597-99c4-9deaaf163659-home\") pod \"auth-test-model-440c60-kserve-5579dbbdbf-2mh9z\" (UID: \"0fcc517d-e223-4597-99c4-9deaaf163659\") " pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" Nov 02 23:38:32.525298 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:32.525281 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fcc517d-e223-4597-99c4-9deaaf163659-tls-certs\") pod \"auth-test-model-440c60-kserve-5579dbbdbf-2mh9z\" (UID: \"0fcc517d-e223-4597-99c4-9deaaf163659\") " pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" Nov 02 23:38:32.525477 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:32.525325 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fcc517d-e223-4597-99c4-9deaaf163659-tmp-dir\") pod \"auth-test-model-440c60-kserve-5579dbbdbf-2mh9z\" (UID: \"0fcc517d-e223-4597-99c4-9deaaf163659\") " pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" Nov 02 23:38:32.525546 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:32.525477 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fcc517d-e223-4597-99c4-9deaaf163659-model-cache\") pod \"auth-test-model-440c60-kserve-5579dbbdbf-2mh9z\" (UID: \"0fcc517d-e223-4597-99c4-9deaaf163659\") " pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" Nov 02 23:38:32.626857 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:32.626772 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fcc517d-e223-4597-99c4-9deaaf163659-model-cache\") pod \"auth-test-model-440c60-kserve-5579dbbdbf-2mh9z\" (UID: \"0fcc517d-e223-4597-99c4-9deaaf163659\") " pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" Nov 02 23:38:32.627008 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:32.626905 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fcc517d-e223-4597-99c4-9deaaf163659-dshm\") pod \"auth-test-model-440c60-kserve-5579dbbdbf-2mh9z\" (UID: \"0fcc517d-e223-4597-99c4-9deaaf163659\") " pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" Nov 02 23:38:32.627008 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:32.626955 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppfmk\" (UniqueName: \"kubernetes.io/projected/0fcc517d-e223-4597-99c4-9deaaf163659-kube-api-access-ppfmk\") pod \"auth-test-model-440c60-kserve-5579dbbdbf-2mh9z\" (UID: \"0fcc517d-e223-4597-99c4-9deaaf163659\") " pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" Nov 02 23:38:32.627109 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:32.627097 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fcc517d-e223-4597-99c4-9deaaf163659-home\") pod \"auth-test-model-440c60-kserve-5579dbbdbf-2mh9z\" (UID: \"0fcc517d-e223-4597-99c4-9deaaf163659\") " pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" Nov 02 23:38:32.627212 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:32.627167 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fcc517d-e223-4597-99c4-9deaaf163659-tls-certs\") pod \"auth-test-model-440c60-kserve-5579dbbdbf-2mh9z\" (UID: \"0fcc517d-e223-4597-99c4-9deaaf163659\") " pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" Nov 02 23:38:32.627275 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:32.627223 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fcc517d-e223-4597-99c4-9deaaf163659-tmp-dir\") pod \"auth-test-model-440c60-kserve-5579dbbdbf-2mh9z\" (UID: \"0fcc517d-e223-4597-99c4-9deaaf163659\") " pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" Nov 02 23:38:32.627275 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:32.627256 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fcc517d-e223-4597-99c4-9deaaf163659-model-cache\") pod \"auth-test-model-440c60-kserve-5579dbbdbf-2mh9z\" (UID: \"0fcc517d-e223-4597-99c4-9deaaf163659\") " pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" Nov 02 23:38:32.627519 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:32.627494 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fcc517d-e223-4597-99c4-9deaaf163659-home\") pod \"auth-test-model-440c60-kserve-5579dbbdbf-2mh9z\" (UID: \"0fcc517d-e223-4597-99c4-9deaaf163659\") " pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" Nov 02 23:38:32.627600 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:32.627575 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fcc517d-e223-4597-99c4-9deaaf163659-tmp-dir\") pod \"auth-test-model-440c60-kserve-5579dbbdbf-2mh9z\" (UID: \"0fcc517d-e223-4597-99c4-9deaaf163659\") " pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" Nov 02 23:38:32.629355 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:32.629332 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fcc517d-e223-4597-99c4-9deaaf163659-dshm\") pod \"auth-test-model-440c60-kserve-5579dbbdbf-2mh9z\" (UID: \"0fcc517d-e223-4597-99c4-9deaaf163659\") " pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" Nov 02 23:38:32.629762 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:32.629740 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fcc517d-e223-4597-99c4-9deaaf163659-tls-certs\") pod \"auth-test-model-440c60-kserve-5579dbbdbf-2mh9z\" (UID: \"0fcc517d-e223-4597-99c4-9deaaf163659\") " pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" Nov 02 23:38:32.693260 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:32.693221 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppfmk\" (UniqueName: \"kubernetes.io/projected/0fcc517d-e223-4597-99c4-9deaaf163659-kube-api-access-ppfmk\") pod \"auth-test-model-440c60-kserve-5579dbbdbf-2mh9z\" (UID: \"0fcc517d-e223-4597-99c4-9deaaf163659\") " pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" Nov 02 23:38:32.985893 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:32.985803 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" Nov 02 23:38:33.368098 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:33.368069 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z"] Nov 02 23:38:33.369267 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:38:33.369236 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fcc517d_e223_4597_99c4_9deaaf163659.slice/crio-f06604e79bf87cd4f88949a40e068d15a9a0184293486552a5f94e0d88e30476 WatchSource:0}: Error finding container f06604e79bf87cd4f88949a40e068d15a9a0184293486552a5f94e0d88e30476: Status 404 returned error can't find the container with id f06604e79bf87cd4f88949a40e068d15a9a0184293486552a5f94e0d88e30476 Nov 02 23:38:34.363176 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:34.363108 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" event={"ID":"0fcc517d-e223-4597-99c4-9deaaf163659","Type":"ContainerStarted","Data":"d8262f6deebc2b694908dd7a7039da01c8ca503c74681186ba4debb59892ae5a"} Nov 02 23:38:34.363399 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:34.363193 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" event={"ID":"0fcc517d-e223-4597-99c4-9deaaf163659","Type":"ContainerStarted","Data":"f06604e79bf87cd4f88949a40e068d15a9a0184293486552a5f94e0d88e30476"} Nov 02 23:38:34.479426 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:34.479374 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" podStartSLOduration=2.479362337 podStartE2EDuration="2.479362337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:38:34.478462905 +0000 UTC m=+3000.716960568" watchObservedRunningTime="2026-11-02 23:38:34.479362337 +0000 UTC m=+3000.717859999" Nov 02 23:38:34.606187 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:34.606151 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 02 23:38:34.635950 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:34.635874 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 23:38:34.693266 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:34.693243 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 02 23:38:34.709961 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:34.709942 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 23:38:34.816835 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:34.816803 5727 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 02 23:38:34.816991 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:34.816876 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" podUID="cabff31e-d3dd-469a-8f1d-f4e8fd22fe18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 02 23:38:34.985920 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:34.985881 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" Nov 02 23:38:35.015843 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.015811 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:38:35.069190 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.069146 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-kube-api-access-jtfz6\" (UniqueName: \"kubernetes.io/projected/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-kube-api-access-jtfz6\") pod \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " Nov 02 23:38:35.069190 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.069191 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istio-podinfo\") pod \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " Nov 02 23:38:35.069473 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.069262 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-workload-socket\") pod \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " Nov 02 23:38:35.069473 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.069295 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-workload-certs\") pod \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " Nov 02 23:38:35.069473 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.069360 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istio-envoy\") pod \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " Nov 02 23:38:35.069473 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.069434 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istio-data\") pod \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " Nov 02 23:38:35.069687 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.069486 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istio-token\" (UniqueName: \"kubernetes.io/projected/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istio-token\") pod \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " Nov 02 23:38:35.069687 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.069531 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-credential-socket\") pod \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " Nov 02 23:38:35.069687 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.069651 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istiod-ca-cert\") pod \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\" (UID: \"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18\") " Nov 02 23:38:35.070407 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.070161 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-workload-socket" pod "cabff31e-d3dd-469a-8f1d-f4e8fd22fe18" (UID: "cabff31e-d3dd-469a-8f1d-f4e8fd22fe18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:35.070407 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.070188 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istio-data" pod "cabff31e-d3dd-469a-8f1d-f4e8fd22fe18" (UID: "cabff31e-d3dd-469a-8f1d-f4e8fd22fe18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:35.070622 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.070430 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-credential-socket" pod "cabff31e-d3dd-469a-8f1d-f4e8fd22fe18" (UID: "cabff31e-d3dd-469a-8f1d-f4e8fd22fe18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:35.070622 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.070454 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istiod-ca-cert" pod "cabff31e-d3dd-469a-8f1d-f4e8fd22fe18" (UID: "cabff31e-d3dd-469a-8f1d-f4e8fd22fe18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:38:35.070912 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.070840 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-workload-certs" pod "cabff31e-d3dd-469a-8f1d-f4e8fd22fe18" (UID: "cabff31e-d3dd-469a-8f1d-f4e8fd22fe18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:35.072358 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.072336 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istio-token" pod "cabff31e-d3dd-469a-8f1d-f4e8fd22fe18" (UID: "cabff31e-d3dd-469a-8f1d-f4e8fd22fe18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:38:35.073507 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.073482 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istio-envoy" pod "cabff31e-d3dd-469a-8f1d-f4e8fd22fe18" (UID: "cabff31e-d3dd-469a-8f1d-f4e8fd22fe18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:35.076155 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.075451 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-kube-api-access-jtfz6" pod "cabff31e-d3dd-469a-8f1d-f4e8fd22fe18" (UID: "cabff31e-d3dd-469a-8f1d-f4e8fd22fe18"). InnerVolumeSpecName "kube-api-access-jtfz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:38:35.076749 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.076714 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istio-podinfo" pod "cabff31e-d3dd-469a-8f1d-f4e8fd22fe18" (UID: "cabff31e-d3dd-469a-8f1d-f4e8fd22fe18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:38:35.171658 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.171572 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtfz6\" (UniqueName: \"kubernetes.io/projected/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-kube-api-access-jtfz6\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.171658 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.171612 5727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istio-podinfo\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.171658 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.171628 5727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-workload-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.171658 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.171643 5727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-workload-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.171658 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.171657 5727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istio-envoy\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.171933 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.171669 5727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istio-data\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.171933 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.171683 5727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istio-token\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.171933 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.171695 5727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-credential-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.171933 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.171708 5727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18-istiod-ca-cert\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.377586 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.377545 5727 generic.go:356] "Generic (PLEG): container finished" podID="cabff31e-d3dd-469a-8f1d-f4e8fd22fe18" containerID="470c9d03cb4a9a50f315a394a7b662107a6ad1cda9b858eb14328283fb826a7b" exitCode=0 Nov 02 23:38:35.377794 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.377640 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" Nov 02 23:38:35.377794 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.377678 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" event={"ID":"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18","Type":"ContainerDied","Data":"470c9d03cb4a9a50f315a394a7b662107a6ad1cda9b858eb14328283fb826a7b"} Nov 02 23:38:35.377794 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.377754 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2" event={"ID":"cabff31e-d3dd-469a-8f1d-f4e8fd22fe18","Type":"ContainerDied","Data":"c466675f3535f18a63599d1afcf1ba9a075b3d2f59cd3fe7c17f52edecae27a7"} Nov 02 23:38:35.377794 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.377783 5727 scope.go:122] "RemoveContainer" containerID="470c9d03cb4a9a50f315a394a7b662107a6ad1cda9b858eb14328283fb826a7b" Nov 02 23:38:35.393488 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.393462 5727 scope.go:122] "RemoveContainer" containerID="470c9d03cb4a9a50f315a394a7b662107a6ad1cda9b858eb14328283fb826a7b" Nov 02 23:38:35.393847 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:38:35.393825 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"470c9d03cb4a9a50f315a394a7b662107a6ad1cda9b858eb14328283fb826a7b\": container with ID starting with 470c9d03cb4a9a50f315a394a7b662107a6ad1cda9b858eb14328283fb826a7b not found: ID does not exist" containerID="470c9d03cb4a9a50f315a394a7b662107a6ad1cda9b858eb14328283fb826a7b" Nov 02 23:38:35.393959 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.393943 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"470c9d03cb4a9a50f315a394a7b662107a6ad1cda9b858eb14328283fb826a7b"} err="failed to get container status \"470c9d03cb4a9a50f315a394a7b662107a6ad1cda9b858eb14328283fb826a7b\": rpc error: code = NotFound desc = could not find container \"470c9d03cb4a9a50f315a394a7b662107a6ad1cda9b858eb14328283fb826a7b\": container with ID starting with 470c9d03cb4a9a50f315a394a7b662107a6ad1cda9b858eb14328283fb826a7b not found: ID does not exist" Nov 02 23:38:35.543829 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.543747 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2"] Nov 02 23:38:35.595651 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.595618 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-z99m2"] Nov 02 23:38:35.887752 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:35.887716 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cabff31e-d3dd-469a-8f1d-f4e8fd22fe18" path="/var/lib/kubelet/pods/cabff31e-d3dd-469a-8f1d-f4e8fd22fe18/volumes" Nov 02 23:38:42.986551 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:42.986516 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" Nov 02 23:38:43.000337 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:43.000313 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" Nov 02 23:38:43.453570 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:43.453538 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" Nov 02 23:38:57.286980 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:57.286949 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz"] Nov 02 23:38:57.287482 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:38:57.287173 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" podUID="465b7605-8ea0-42da-a879-d129b29ddf3f" containerName="istio-proxy" containerID="cri-o://ce0ad02656d87501ef9ebe95999a783f9ba33258fe7cb52ed9eeb9e8e6cb185b" gracePeriod=30 Nov 02 23:38:57.864056 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:38:57.864000 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:39:02.577008 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.576972 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:39:02.661074 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.661046 5727 generic.go:356] "Generic (PLEG): container finished" podID="465b7605-8ea0-42da-a879-d129b29ddf3f" containerID="ce0ad02656d87501ef9ebe95999a783f9ba33258fe7cb52ed9eeb9e8e6cb185b" exitCode=0 Nov 02 23:39:02.661255 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.661106 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" Nov 02 23:39:02.661255 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.661152 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" event={"ID":"465b7605-8ea0-42da-a879-d129b29ddf3f","Type":"ContainerDied","Data":"ce0ad02656d87501ef9ebe95999a783f9ba33258fe7cb52ed9eeb9e8e6cb185b"} Nov 02 23:39:02.661255 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.661219 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz" event={"ID":"465b7605-8ea0-42da-a879-d129b29ddf3f","Type":"ContainerDied","Data":"5dd3b3b3647f0889626544570d101975ce1a38de4fb4d9669ac9bb3a913a63aa"} Nov 02 23:39:02.661364 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.661279 5727 scope.go:122] "RemoveContainer" containerID="ce0ad02656d87501ef9ebe95999a783f9ba33258fe7cb52ed9eeb9e8e6cb185b" Nov 02 23:39:02.675392 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.675370 5727 scope.go:122] "RemoveContainer" containerID="ce0ad02656d87501ef9ebe95999a783f9ba33258fe7cb52ed9eeb9e8e6cb185b" Nov 02 23:39:02.675734 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:39:02.675696 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce0ad02656d87501ef9ebe95999a783f9ba33258fe7cb52ed9eeb9e8e6cb185b\": container with ID starting with ce0ad02656d87501ef9ebe95999a783f9ba33258fe7cb52ed9eeb9e8e6cb185b not found: ID does not exist" containerID="ce0ad02656d87501ef9ebe95999a783f9ba33258fe7cb52ed9eeb9e8e6cb185b" Nov 02 23:39:02.675806 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.675733 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce0ad02656d87501ef9ebe95999a783f9ba33258fe7cb52ed9eeb9e8e6cb185b"} err="failed to get container status \"ce0ad02656d87501ef9ebe95999a783f9ba33258fe7cb52ed9eeb9e8e6cb185b\": rpc error: code = NotFound desc = could not find container \"ce0ad02656d87501ef9ebe95999a783f9ba33258fe7cb52ed9eeb9e8e6cb185b\": container with ID starting with ce0ad02656d87501ef9ebe95999a783f9ba33258fe7cb52ed9eeb9e8e6cb185b not found: ID does not exist" Nov 02 23:39:02.746476 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.746429 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/465b7605-8ea0-42da-a879-d129b29ddf3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/465b7605-8ea0-42da-a879-d129b29ddf3f-istio-podinfo\") pod \"465b7605-8ea0-42da-a879-d129b29ddf3f\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " Nov 02 23:39:02.746720 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.746530 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-workload-certs\") pod \"465b7605-8ea0-42da-a879-d129b29ddf3f\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " Nov 02 23:39:02.746720 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.746652 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-istio-data\") pod \"465b7605-8ea0-42da-a879-d129b29ddf3f\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " Nov 02 23:39:02.746720 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.746707 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/465b7605-8ea0-42da-a879-d129b29ddf3f-kube-api-access-c69vd\" (UniqueName: \"kubernetes.io/projected/465b7605-8ea0-42da-a879-d129b29ddf3f-kube-api-access-c69vd\") pod \"465b7605-8ea0-42da-a879-d129b29ddf3f\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " Nov 02 23:39:02.746961 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.746895 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-workload-certs" pod "465b7605-8ea0-42da-a879-d129b29ddf3f" (UID: "465b7605-8ea0-42da-a879-d129b29ddf3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:39:02.747036 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.747002 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/465b7605-8ea0-42da-a879-d129b29ddf3f-istio-token\" (UniqueName: \"kubernetes.io/projected/465b7605-8ea0-42da-a879-d129b29ddf3f-istio-token\") pod \"465b7605-8ea0-42da-a879-d129b29ddf3f\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " Nov 02 23:39:02.747110 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.747094 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-credential-socket\") pod \"465b7605-8ea0-42da-a879-d129b29ddf3f\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " Nov 02 23:39:02.747213 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.747165 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-istio-envoy\") pod \"465b7605-8ea0-42da-a879-d129b29ddf3f\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " Nov 02 23:39:02.747275 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.747248 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-workload-socket\") pod \"465b7605-8ea0-42da-a879-d129b29ddf3f\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " Nov 02 23:39:02.747338 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.747322 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/465b7605-8ea0-42da-a879-d129b29ddf3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/465b7605-8ea0-42da-a879-d129b29ddf3f-istiod-ca-cert\") pod \"465b7605-8ea0-42da-a879-d129b29ddf3f\" (UID: \"465b7605-8ea0-42da-a879-d129b29ddf3f\") " Nov 02 23:39:02.747444 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.747008 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-istio-data" pod "465b7605-8ea0-42da-a879-d129b29ddf3f" (UID: "465b7605-8ea0-42da-a879-d129b29ddf3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:39:02.747530 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.747435 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-credential-socket" pod "465b7605-8ea0-42da-a879-d129b29ddf3f" (UID: "465b7605-8ea0-42da-a879-d129b29ddf3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:39:02.748249 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.748180 5727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-credential-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:39:02.748249 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.748191 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-workload-socket" pod "465b7605-8ea0-42da-a879-d129b29ddf3f" (UID: "465b7605-8ea0-42da-a879-d129b29ddf3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:39:02.748249 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.748208 5727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-workload-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:39:02.748249 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.748225 5727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-istio-data\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:39:02.748555 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.748280 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/465b7605-8ea0-42da-a879-d129b29ddf3f-istiod-ca-cert" pod "465b7605-8ea0-42da-a879-d129b29ddf3f" (UID: "465b7605-8ea0-42da-a879-d129b29ddf3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:39:02.749862 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.749831 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/465b7605-8ea0-42da-a879-d129b29ddf3f-kube-api-access-c69vd" pod "465b7605-8ea0-42da-a879-d129b29ddf3f" (UID: "465b7605-8ea0-42da-a879-d129b29ddf3f"). InnerVolumeSpecName "kube-api-access-c69vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:39:02.750220 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.750063 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/465b7605-8ea0-42da-a879-d129b29ddf3f-istio-podinfo" pod "465b7605-8ea0-42da-a879-d129b29ddf3f" (UID: "465b7605-8ea0-42da-a879-d129b29ddf3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:39:02.750220 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.750188 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/465b7605-8ea0-42da-a879-d129b29ddf3f-istio-token" pod "465b7605-8ea0-42da-a879-d129b29ddf3f" (UID: "465b7605-8ea0-42da-a879-d129b29ddf3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:39:02.750379 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.750339 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-istio-envoy" pod "465b7605-8ea0-42da-a879-d129b29ddf3f" (UID: "465b7605-8ea0-42da-a879-d129b29ddf3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:39:02.849266 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.849200 5727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/465b7605-8ea0-42da-a879-d129b29ddf3f-istio-token\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:39:02.849266 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.849227 5727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-istio-envoy\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:39:02.849266 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.849237 5727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/465b7605-8ea0-42da-a879-d129b29ddf3f-workload-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:39:02.849266 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.849246 5727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/465b7605-8ea0-42da-a879-d129b29ddf3f-istiod-ca-cert\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:39:02.849266 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.849254 5727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/465b7605-8ea0-42da-a879-d129b29ddf3f-istio-podinfo\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:39:02.849266 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:02.849263 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c69vd\" (UniqueName: \"kubernetes.io/projected/465b7605-8ea0-42da-a879-d129b29ddf3f-kube-api-access-c69vd\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:39:03.089333 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:03.088004 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz"] Nov 02 23:39:03.100855 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:03.100749 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-hh7jz"] Nov 02 23:39:03.885629 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:39:03.885597 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="465b7605-8ea0-42da-a879-d129b29ddf3f" path="/var/lib/kubelet/pods/465b7605-8ea0-42da-a879-d129b29ddf3f/volumes" Nov 02 23:39:27.845807 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:39:27.845766 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:39:57.846553 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:39:57.846512 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:40:07.361469 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:07.361440 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz"] Nov 02 23:40:07.362091 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:07.362076 5727 cpu_manager.go:414] "containerMap: removing container" podUID="cabff31e-d3dd-469a-8f1d-f4e8fd22fe18" containerName="istio-proxy" Nov 02 23:40:07.362154 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:07.362091 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cabff31e-d3dd-469a-8f1d-f4e8fd22fe18" containerName="istio-proxy" Nov 02 23:40:07.362193 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:07.362155 5727 cpu_manager.go:414] "containerMap: removing container" podUID="465b7605-8ea0-42da-a879-d129b29ddf3f" containerName="istio-proxy" Nov 02 23:40:07.362193 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:07.362161 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="465b7605-8ea0-42da-a879-d129b29ddf3f" containerName="istio-proxy" Nov 02 23:40:07.362282 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:07.362271 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="465b7605-8ea0-42da-a879-d129b29ddf3f" containerName="istio-proxy" Nov 02 23:40:07.362316 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:07.362289 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="cabff31e-d3dd-469a-8f1d-f4e8fd22fe18" containerName="istio-proxy" Nov 02 23:40:07.365793 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:07.365778 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz" Nov 02 23:40:07.467702 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:07.467665 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-915ccb\"/\"kube-root-ca.crt\"" Nov 02 23:40:07.467888 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:07.467708 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-915ccb\"/\"openshift-service-ca.crt\"" Nov 02 23:40:07.467888 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:07.467708 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-915ccb\"/\"default-dockercfg-jzg7g\"" Nov 02 23:40:07.468102 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:07.468064 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-915ccb\"/\"auth-test-model-915ccb-kserve-self-signed-certs\"" Nov 02 23:40:07.471742 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:07.471720 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz"] Nov 02 23:40:07.540730 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:07.540696 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaa5326f-dbfd-48cd-bf98-830703edb686-tmp-dir\") pod \"auth-test-model-915ccb-kserve-86f59f45d5-6crmz\" (UID: \"aaa5326f-dbfd-48cd-bf98-830703edb686\") " pod="ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz" Nov 02 23:40:07.540911 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:07.540862 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aaa5326f-dbfd-48cd-bf98-830703edb686-dshm\") pod \"auth-test-model-915ccb-kserve-86f59f45d5-6crmz\" (UID: \"aaa5326f-dbfd-48cd-bf98-830703edb686\") " pod="ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz" Nov 02 23:40:07.540950 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:07.540912 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaa5326f-dbfd-48cd-bf98-830703edb686-tls-certs\") pod \"auth-test-model-915ccb-kserve-86f59f45d5-6crmz\" (UID: \"aaa5326f-dbfd-48cd-bf98-830703edb686\") " pod="ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz" Nov 02 23:40:07.541010 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:07.540996 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aaa5326f-dbfd-48cd-bf98-830703edb686-home\") pod \"auth-test-model-915ccb-kserve-86f59f45d5-6crmz\" (UID: \"aaa5326f-dbfd-48cd-bf98-830703edb686\") " pod="ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz" Nov 02 23:40:07.541056 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:07.541017 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaa5326f-dbfd-48cd-bf98-830703edb686-model-cache\") pod \"auth-test-model-915ccb-kserve-86f59f45d5-6crmz\" (UID: \"aaa5326f-dbfd-48cd-bf98-830703edb686\") " pod="ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz" Nov 02 23:40:07.541056 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:07.541037 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcjdq\" (UniqueName: \"kubernetes.io/projected/aaa5326f-dbfd-48cd-bf98-830703edb686-kube-api-access-fcjdq\") pod \"auth-test-model-915ccb-kserve-86f59f45d5-6crmz\" (UID: \"aaa5326f-dbfd-48cd-bf98-830703edb686\") " pod="ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz" Nov 02 23:40:07.642984 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:07.642899 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aaa5326f-dbfd-48cd-bf98-830703edb686-dshm\") pod \"auth-test-model-915ccb-kserve-86f59f45d5-6crmz\" (UID: \"aaa5326f-dbfd-48cd-bf98-830703edb686\") " pod="ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz" Nov 02 23:40:07.642984 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:07.642961 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaa5326f-dbfd-48cd-bf98-830703edb686-tls-certs\") pod \"auth-test-model-915ccb-kserve-86f59f45d5-6crmz\" (UID: \"aaa5326f-dbfd-48cd-bf98-830703edb686\") " pod="ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz" Nov 02 23:40:07.643268 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:07.642994 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aaa5326f-dbfd-48cd-bf98-830703edb686-home\") pod \"auth-test-model-915ccb-kserve-86f59f45d5-6crmz\" (UID: \"aaa5326f-dbfd-48cd-bf98-830703edb686\") " pod="ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz" Nov 02 23:40:07.643268 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:07.643010 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaa5326f-dbfd-48cd-bf98-830703edb686-model-cache\") pod \"auth-test-model-915ccb-kserve-86f59f45d5-6crmz\" (UID: \"aaa5326f-dbfd-48cd-bf98-830703edb686\") " pod="ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz" Nov 02 23:40:07.643268 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:07.643191 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcjdq\" (UniqueName: \"kubernetes.io/projected/aaa5326f-dbfd-48cd-bf98-830703edb686-kube-api-access-fcjdq\") pod \"auth-test-model-915ccb-kserve-86f59f45d5-6crmz\" (UID: \"aaa5326f-dbfd-48cd-bf98-830703edb686\") " pod="ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz" Nov 02 23:40:07.643504 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:07.643309 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaa5326f-dbfd-48cd-bf98-830703edb686-tmp-dir\") pod \"auth-test-model-915ccb-kserve-86f59f45d5-6crmz\" (UID: \"aaa5326f-dbfd-48cd-bf98-830703edb686\") " pod="ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz" Nov 02 23:40:07.643559 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:07.643523 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaa5326f-dbfd-48cd-bf98-830703edb686-model-cache\") pod \"auth-test-model-915ccb-kserve-86f59f45d5-6crmz\" (UID: \"aaa5326f-dbfd-48cd-bf98-830703edb686\") " pod="ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz" Nov 02 23:40:07.643616 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:07.643564 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aaa5326f-dbfd-48cd-bf98-830703edb686-home\") pod \"auth-test-model-915ccb-kserve-86f59f45d5-6crmz\" (UID: \"aaa5326f-dbfd-48cd-bf98-830703edb686\") " pod="ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz" Nov 02 23:40:07.643755 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:07.643732 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaa5326f-dbfd-48cd-bf98-830703edb686-tmp-dir\") pod \"auth-test-model-915ccb-kserve-86f59f45d5-6crmz\" (UID: \"aaa5326f-dbfd-48cd-bf98-830703edb686\") " pod="ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz" Nov 02 23:40:07.645632 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:07.645611 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aaa5326f-dbfd-48cd-bf98-830703edb686-dshm\") pod \"auth-test-model-915ccb-kserve-86f59f45d5-6crmz\" (UID: \"aaa5326f-dbfd-48cd-bf98-830703edb686\") " pod="ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz" Nov 02 23:40:07.646018 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:07.645996 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaa5326f-dbfd-48cd-bf98-830703edb686-tls-certs\") pod \"auth-test-model-915ccb-kserve-86f59f45d5-6crmz\" (UID: \"aaa5326f-dbfd-48cd-bf98-830703edb686\") " pod="ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz" Nov 02 23:40:07.706026 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:07.705995 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcjdq\" (UniqueName: \"kubernetes.io/projected/aaa5326f-dbfd-48cd-bf98-830703edb686-kube-api-access-fcjdq\") pod \"auth-test-model-915ccb-kserve-86f59f45d5-6crmz\" (UID: \"aaa5326f-dbfd-48cd-bf98-830703edb686\") " pod="ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz" Nov 02 23:40:07.978809 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:07.978732 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz" Nov 02 23:40:08.364243 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:08.364219 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz"] Nov 02 23:40:08.365491 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:40:08.365459 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaaa5326f_dbfd_48cd_bf98_830703edb686.slice/crio-677118009ab076520579e980235f2b88d7fb12b99c676c7ab0e1260402d22c82 WatchSource:0}: Error finding container 677118009ab076520579e980235f2b88d7fb12b99c676c7ab0e1260402d22c82: Status 404 returned error can't find the container with id 677118009ab076520579e980235f2b88d7fb12b99c676c7ab0e1260402d22c82 Nov 02 23:40:09.211694 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:09.211656 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz" event={"ID":"aaa5326f-dbfd-48cd-bf98-830703edb686","Type":"ContainerStarted","Data":"e0863f5cf28656e4005ad8e5086c41aa6d85e2c7d8879252af55a02870e859c2"} Nov 02 23:40:09.211694 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:09.211707 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz" event={"ID":"aaa5326f-dbfd-48cd-bf98-830703edb686","Type":"ContainerStarted","Data":"677118009ab076520579e980235f2b88d7fb12b99c676c7ab0e1260402d22c82"} Nov 02 23:40:09.367163 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:09.367062 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz" podStartSLOduration=2.3670443 podStartE2EDuration="2.3670443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:40:09.361369019 +0000 UTC m=+3095.599866703" watchObservedRunningTime="2026-11-02 23:40:09.3670443 +0000 UTC m=+3095.605541964" Nov 02 23:40:09.979914 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:09.979873 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz" Nov 02 23:40:17.979304 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:17.979266 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz" Nov 02 23:40:17.997743 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:17.997713 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz" Nov 02 23:40:18.307536 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:18.307447 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz" Nov 02 23:40:27.847591 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:40:27.847551 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:40:40.191070 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:40.191034 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a5d6d8-568d9db995-7v5c4"] Nov 02 23:40:45.191529 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:45.191477 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a5d6d8-568d9db995-7v5c4" podUID="4ec013db-b396-4d4e-8d00-5cf3f957b9c8" containerName="payload-pre-processing" containerID="cri-o://f5aa4d27515214562f629fe968a46f9c84e6b064c8c9fe7f3ef778ad183cdf74" gracePeriod=25 Nov 02 23:40:45.489350 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:45.489325 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a5d6d8-568d9db995-7v5c4" Nov 02 23:40:45.547100 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:45.547070 5727 generic.go:356] "Generic (PLEG): container finished" podID="4ec013db-b396-4d4e-8d00-5cf3f957b9c8" containerID="f5aa4d27515214562f629fe968a46f9c84e6b064c8c9fe7f3ef778ad183cdf74" exitCode=0 Nov 02 23:40:45.547325 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:45.547213 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a5d6d8-568d9db995-7v5c4" Nov 02 23:40:45.547325 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:45.547248 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a5d6d8-568d9db995-7v5c4" event={"ID":"4ec013db-b396-4d4e-8d00-5cf3f957b9c8","Type":"ContainerDied","Data":"f5aa4d27515214562f629fe968a46f9c84e6b064c8c9fe7f3ef778ad183cdf74"} Nov 02 23:40:45.547325 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:45.547310 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a5d6d8-568d9db995-7v5c4" event={"ID":"4ec013db-b396-4d4e-8d00-5cf3f957b9c8","Type":"ContainerDied","Data":"52803860e19b08032228f729e64ceb003071c985cd78f5cc552f3d37f900017b"} Nov 02 23:40:45.547493 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:45.547343 5727 scope.go:122] "RemoveContainer" containerID="f5aa4d27515214562f629fe968a46f9c84e6b064c8c9fe7f3ef778ad183cdf74" Nov 02 23:40:45.560440 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:45.560421 5727 scope.go:122] "RemoveContainer" containerID="f5aa4d27515214562f629fe968a46f9c84e6b064c8c9fe7f3ef778ad183cdf74" Nov 02 23:40:45.560536 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:45.560517 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ec013db-b396-4d4e-8d00-5cf3f957b9c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ec013db-b396-4d4e-8d00-5cf3f957b9c8-plugins-config-volume\") pod \"4ec013db-b396-4d4e-8d00-5cf3f957b9c8\" (UID: \"4ec013db-b396-4d4e-8d00-5cf3f957b9c8\") " Nov 02 23:40:45.560667 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:45.560578 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ec013db-b396-4d4e-8d00-5cf3f957b9c8-kube-api-access-sw486\" (UniqueName: \"kubernetes.io/projected/4ec013db-b396-4d4e-8d00-5cf3f957b9c8-kube-api-access-sw486\") pod \"4ec013db-b396-4d4e-8d00-5cf3f957b9c8\" (UID: \"4ec013db-b396-4d4e-8d00-5cf3f957b9c8\") " Nov 02 23:40:45.560796 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:40:45.560768 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5aa4d27515214562f629fe968a46f9c84e6b064c8c9fe7f3ef778ad183cdf74\": container with ID starting with f5aa4d27515214562f629fe968a46f9c84e6b064c8c9fe7f3ef778ad183cdf74 not found: ID does not exist" containerID="f5aa4d27515214562f629fe968a46f9c84e6b064c8c9fe7f3ef778ad183cdf74" Nov 02 23:40:45.560892 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:45.560810 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5aa4d27515214562f629fe968a46f9c84e6b064c8c9fe7f3ef778ad183cdf74"} err="failed to get container status \"f5aa4d27515214562f629fe968a46f9c84e6b064c8c9fe7f3ef778ad183cdf74\": rpc error: code = NotFound desc = could not find container \"f5aa4d27515214562f629fe968a46f9c84e6b064c8c9fe7f3ef778ad183cdf74\": container with ID starting with f5aa4d27515214562f629fe968a46f9c84e6b064c8c9fe7f3ef778ad183cdf74 not found: ID does not exist" Nov 02 23:40:45.561003 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:45.560976 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ec013db-b396-4d4e-8d00-5cf3f957b9c8-plugins-config-volume" pod "4ec013db-b396-4d4e-8d00-5cf3f957b9c8" (UID: "4ec013db-b396-4d4e-8d00-5cf3f957b9c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:40:45.561162 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:45.561117 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ec013db-b396-4d4e-8d00-5cf3f957b9c8-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:45.563315 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:45.563293 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ec013db-b396-4d4e-8d00-5cf3f957b9c8-kube-api-access-sw486" pod "4ec013db-b396-4d4e-8d00-5cf3f957b9c8" (UID: "4ec013db-b396-4d4e-8d00-5cf3f957b9c8"). InnerVolumeSpecName "kube-api-access-sw486". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:40:45.662528 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:45.662485 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw486\" (UniqueName: \"kubernetes.io/projected/4ec013db-b396-4d4e-8d00-5cf3f957b9c8-kube-api-access-sw486\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:45.968277 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:45.968238 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a5d6d8-568d9db995-7v5c4"] Nov 02 23:40:45.973853 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:45.973828 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a5d6d8-568d9db995-7v5c4"] Nov 02 23:40:47.885628 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:40:47.885592 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ec013db-b396-4d4e-8d00-5cf3f957b9c8" path="/var/lib/kubelet/pods/4ec013db-b396-4d4e-8d00-5cf3f957b9c8/volumes" Nov 02 23:40:57.847075 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:40:57.847030 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:41:27.847390 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:41:27.847352 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:41:31.079721 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:31.079686 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9"] Nov 02 23:41:31.080266 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:31.079880 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" podUID="bf35636b-a796-4dfc-83af-9f052a649ae3" containerName="istio-proxy" containerID="cri-o://1c347628a4e5641dbf245979625e854d9238469958709bb5b38551e6c30f3672" gracePeriod=30 Nov 02 23:41:36.362767 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.362695 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:41:36.433081 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.433043 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-istio-envoy\") pod \"bf35636b-a796-4dfc-83af-9f052a649ae3\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " Nov 02 23:41:36.433273 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.433169 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf35636b-a796-4dfc-83af-9f052a649ae3-istio-token\" (UniqueName: \"kubernetes.io/projected/bf35636b-a796-4dfc-83af-9f052a649ae3-istio-token\") pod \"bf35636b-a796-4dfc-83af-9f052a649ae3\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " Nov 02 23:41:36.433339 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.433274 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf35636b-a796-4dfc-83af-9f052a649ae3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf35636b-a796-4dfc-83af-9f052a649ae3-istiod-ca-cert\") pod \"bf35636b-a796-4dfc-83af-9f052a649ae3\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " Nov 02 23:41:36.433339 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.433316 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf35636b-a796-4dfc-83af-9f052a649ae3-kube-api-access-vvmzv\" (UniqueName: \"kubernetes.io/projected/bf35636b-a796-4dfc-83af-9f052a649ae3-kube-api-access-vvmzv\") pod \"bf35636b-a796-4dfc-83af-9f052a649ae3\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " Nov 02 23:41:36.433444 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.433373 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-credential-socket\") pod \"bf35636b-a796-4dfc-83af-9f052a649ae3\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " Nov 02 23:41:36.433444 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.433403 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf35636b-a796-4dfc-83af-9f052a649ae3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf35636b-a796-4dfc-83af-9f052a649ae3-istio-podinfo\") pod \"bf35636b-a796-4dfc-83af-9f052a649ae3\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " Nov 02 23:41:36.433558 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.433472 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-istio-data\") pod \"bf35636b-a796-4dfc-83af-9f052a649ae3\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " Nov 02 23:41:36.433558 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.433528 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-workload-socket\") pod \"bf35636b-a796-4dfc-83af-9f052a649ae3\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " Nov 02 23:41:36.433669 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.433594 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-workload-certs\") pod \"bf35636b-a796-4dfc-83af-9f052a649ae3\" (UID: \"bf35636b-a796-4dfc-83af-9f052a649ae3\") " Nov 02 23:41:36.433669 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.433600 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-credential-socket" pod "bf35636b-a796-4dfc-83af-9f052a649ae3" (UID: "bf35636b-a796-4dfc-83af-9f052a649ae3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:41:36.433785 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.433676 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf35636b-a796-4dfc-83af-9f052a649ae3-istiod-ca-cert" pod "bf35636b-a796-4dfc-83af-9f052a649ae3" (UID: "bf35636b-a796-4dfc-83af-9f052a649ae3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:41:36.433846 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.433791 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-istio-data" pod "bf35636b-a796-4dfc-83af-9f052a649ae3" (UID: "bf35636b-a796-4dfc-83af-9f052a649ae3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:41:36.433993 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.433968 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-workload-socket" pod "bf35636b-a796-4dfc-83af-9f052a649ae3" (UID: "bf35636b-a796-4dfc-83af-9f052a649ae3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:41:36.433993 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.433982 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-workload-certs" pod "bf35636b-a796-4dfc-83af-9f052a649ae3" (UID: "bf35636b-a796-4dfc-83af-9f052a649ae3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:41:36.434282 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.434262 5727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-istio-data\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:36.434337 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.434291 5727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-workload-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:36.434337 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.434306 5727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-workload-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:36.434337 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.434320 5727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf35636b-a796-4dfc-83af-9f052a649ae3-istiod-ca-cert\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:36.434337 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.434334 5727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-credential-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:36.435675 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.435642 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf35636b-a796-4dfc-83af-9f052a649ae3-istio-token" pod "bf35636b-a796-4dfc-83af-9f052a649ae3" (UID: "bf35636b-a796-4dfc-83af-9f052a649ae3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:41:36.436059 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.436036 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-istio-envoy" pod "bf35636b-a796-4dfc-83af-9f052a649ae3" (UID: "bf35636b-a796-4dfc-83af-9f052a649ae3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:41:36.436293 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.436271 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf35636b-a796-4dfc-83af-9f052a649ae3-istio-podinfo" pod "bf35636b-a796-4dfc-83af-9f052a649ae3" (UID: "bf35636b-a796-4dfc-83af-9f052a649ae3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:41:36.436370 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.436294 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf35636b-a796-4dfc-83af-9f052a649ae3-kube-api-access-vvmzv" pod "bf35636b-a796-4dfc-83af-9f052a649ae3" (UID: "bf35636b-a796-4dfc-83af-9f052a649ae3"). InnerVolumeSpecName "kube-api-access-vvmzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:41:36.536334 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.536300 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvmzv\" (UniqueName: \"kubernetes.io/projected/bf35636b-a796-4dfc-83af-9f052a649ae3-kube-api-access-vvmzv\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:36.536334 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.536330 5727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf35636b-a796-4dfc-83af-9f052a649ae3-istio-podinfo\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:36.536334 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.536341 5727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf35636b-a796-4dfc-83af-9f052a649ae3-istio-envoy\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:36.536591 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.536350 5727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf35636b-a796-4dfc-83af-9f052a649ae3-istio-token\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:36.758778 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.758693 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4cdec0-ccc48bdc7-dkvtg"] Nov 02 23:41:36.868788 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.868754 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4cdec0-65f74dfb55-6896h"] Nov 02 23:41:36.966384 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.966348 5727 generic.go:356] "Generic (PLEG): container finished" podID="bf35636b-a796-4dfc-83af-9f052a649ae3" containerID="1c347628a4e5641dbf245979625e854d9238469958709bb5b38551e6c30f3672" exitCode=0 Nov 02 23:41:36.966586 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.966412 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" Nov 02 23:41:36.966586 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.966434 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" event={"ID":"bf35636b-a796-4dfc-83af-9f052a649ae3","Type":"ContainerDied","Data":"1c347628a4e5641dbf245979625e854d9238469958709bb5b38551e6c30f3672"} Nov 02 23:41:36.966586 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.966495 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9" event={"ID":"bf35636b-a796-4dfc-83af-9f052a649ae3","Type":"ContainerDied","Data":"56794ef6077543ded36d7388cc240a7fa62ebed410eb7ce7cc81f6a17ec02ef7"} Nov 02 23:41:36.966586 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.966524 5727 scope.go:122] "RemoveContainer" containerID="1c347628a4e5641dbf245979625e854d9238469958709bb5b38551e6c30f3672" Nov 02 23:41:36.984072 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.984056 5727 scope.go:122] "RemoveContainer" containerID="1c347628a4e5641dbf245979625e854d9238469958709bb5b38551e6c30f3672" Nov 02 23:41:36.984375 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:41:36.984357 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c347628a4e5641dbf245979625e854d9238469958709bb5b38551e6c30f3672\": container with ID starting with 1c347628a4e5641dbf245979625e854d9238469958709bb5b38551e6c30f3672 not found: ID does not exist" containerID="1c347628a4e5641dbf245979625e854d9238469958709bb5b38551e6c30f3672" Nov 02 23:41:36.984463 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:36.984381 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c347628a4e5641dbf245979625e854d9238469958709bb5b38551e6c30f3672"} err="failed to get container status \"1c347628a4e5641dbf245979625e854d9238469958709bb5b38551e6c30f3672\": rpc error: code = NotFound desc = could not find container \"1c347628a4e5641dbf245979625e854d9238469958709bb5b38551e6c30f3672\": container with ID starting with 1c347628a4e5641dbf245979625e854d9238469958709bb5b38551e6c30f3672 not found: ID does not exist" Nov 02 23:41:37.104251 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:37.104219 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9"] Nov 02 23:41:37.109001 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:37.108976 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-cgnj9"] Nov 02 23:41:37.885838 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:37.885805 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf35636b-a796-4dfc-83af-9f052a649ae3" path="/var/lib/kubelet/pods/bf35636b-a796-4dfc-83af-9f052a649ae3/volumes" Nov 02 23:41:41.759241 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:41.759188 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cdec0-ccc48bdc7-dkvtg" podUID="f0b5de66-8158-4fb5-b340-d7a5b85f2766" containerName="payload-processing" containerID="cri-o://d13dd821c2a4626b5ba67c049704e1fc36804e3c3841bb4ceb2167df350ff3ac" gracePeriod=25 Nov 02 23:41:41.843917 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:41:41.843872 5727 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b5de66_8158_4fb5_b340_d7a5b85f2766.slice/crio-conmon-d13dd821c2a4626b5ba67c049704e1fc36804e3c3841bb4ceb2167df350ff3ac.scope\": RecentStats: unable to find data in memory cache]" Nov 02 23:41:41.869518 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:41.869480 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cdec0-65f74dfb55-6896h" podUID="7abe7663-f846-4499-b6d4-0c85ab2a148d" containerName="payload-pre-processing" containerID="cri-o://828b080ad38ee8e135f44af3bb3e58e4f44afea6f622c7d6b0611f0aaca804f9" gracePeriod=25 Nov 02 23:41:42.016796 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:42.016758 5727 generic.go:356] "Generic (PLEG): container finished" podID="7abe7663-f846-4499-b6d4-0c85ab2a148d" containerID="828b080ad38ee8e135f44af3bb3e58e4f44afea6f622c7d6b0611f0aaca804f9" exitCode=0 Nov 02 23:41:42.016945 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:42.016812 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cdec0-65f74dfb55-6896h" event={"ID":"7abe7663-f846-4499-b6d4-0c85ab2a148d","Type":"ContainerDied","Data":"828b080ad38ee8e135f44af3bb3e58e4f44afea6f622c7d6b0611f0aaca804f9"} Nov 02 23:41:42.019302 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:42.019271 5727 generic.go:356] "Generic (PLEG): container finished" podID="f0b5de66-8158-4fb5-b340-d7a5b85f2766" containerID="d13dd821c2a4626b5ba67c049704e1fc36804e3c3841bb4ceb2167df350ff3ac" exitCode=0 Nov 02 23:41:42.019477 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:42.019366 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cdec0-ccc48bdc7-dkvtg" event={"ID":"f0b5de66-8158-4fb5-b340-d7a5b85f2766","Type":"ContainerDied","Data":"d13dd821c2a4626b5ba67c049704e1fc36804e3c3841bb4ceb2167df350ff3ac"} Nov 02 23:41:42.019477 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:42.019410 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cdec0-ccc48bdc7-dkvtg" event={"ID":"f0b5de66-8158-4fb5-b340-d7a5b85f2766","Type":"ContainerDied","Data":"09735015aa76c39a76abdea3449db8cce0ea4633d3da1d8de83dfac07fbd24b6"} Nov 02 23:41:42.019477 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:42.019434 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09735015aa76c39a76abdea3449db8cce0ea4633d3da1d8de83dfac07fbd24b6" Nov 02 23:41:42.039942 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:42.039639 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cdec0-ccc48bdc7-dkvtg" Nov 02 23:41:42.151862 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:42.151838 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0b5de66-8158-4fb5-b340-d7a5b85f2766-kube-api-access-tgggw\" (UniqueName: \"kubernetes.io/projected/f0b5de66-8158-4fb5-b340-d7a5b85f2766-kube-api-access-tgggw\") pod \"f0b5de66-8158-4fb5-b340-d7a5b85f2766\" (UID: \"f0b5de66-8158-4fb5-b340-d7a5b85f2766\") " Nov 02 23:41:42.152030 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:42.151879 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0b5de66-8158-4fb5-b340-d7a5b85f2766-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b5de66-8158-4fb5-b340-d7a5b85f2766-plugins-config-volume\") pod \"f0b5de66-8158-4fb5-b340-d7a5b85f2766\" (UID: \"f0b5de66-8158-4fb5-b340-d7a5b85f2766\") " Nov 02 23:41:42.152299 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:42.152270 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0b5de66-8158-4fb5-b340-d7a5b85f2766-plugins-config-volume" pod "f0b5de66-8158-4fb5-b340-d7a5b85f2766" (UID: "f0b5de66-8158-4fb5-b340-d7a5b85f2766"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:41:42.154142 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:42.154106 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0b5de66-8158-4fb5-b340-d7a5b85f2766-kube-api-access-tgggw" pod "f0b5de66-8158-4fb5-b340-d7a5b85f2766" (UID: "f0b5de66-8158-4fb5-b340-d7a5b85f2766"). InnerVolumeSpecName "kube-api-access-tgggw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:41:42.158598 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:42.158572 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cdec0-65f74dfb55-6896h" Nov 02 23:41:42.253668 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:42.253633 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgggw\" (UniqueName: \"kubernetes.io/projected/f0b5de66-8158-4fb5-b340-d7a5b85f2766-kube-api-access-tgggw\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:42.253668 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:42.253661 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b5de66-8158-4fb5-b340-d7a5b85f2766-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:42.354294 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:42.354255 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7abe7663-f846-4499-b6d4-0c85ab2a148d-kube-api-access-mqjkn\" (UniqueName: \"kubernetes.io/projected/7abe7663-f846-4499-b6d4-0c85ab2a148d-kube-api-access-mqjkn\") pod \"7abe7663-f846-4499-b6d4-0c85ab2a148d\" (UID: \"7abe7663-f846-4499-b6d4-0c85ab2a148d\") " Nov 02 23:41:42.354515 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:42.354313 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7abe7663-f846-4499-b6d4-0c85ab2a148d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7abe7663-f846-4499-b6d4-0c85ab2a148d-plugins-config-volume\") pod \"7abe7663-f846-4499-b6d4-0c85ab2a148d\" (UID: \"7abe7663-f846-4499-b6d4-0c85ab2a148d\") " Nov 02 23:41:42.354760 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:42.354735 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7abe7663-f846-4499-b6d4-0c85ab2a148d-plugins-config-volume" pod "7abe7663-f846-4499-b6d4-0c85ab2a148d" (UID: "7abe7663-f846-4499-b6d4-0c85ab2a148d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:41:42.356681 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:42.356659 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7abe7663-f846-4499-b6d4-0c85ab2a148d-kube-api-access-mqjkn" pod "7abe7663-f846-4499-b6d4-0c85ab2a148d" (UID: "7abe7663-f846-4499-b6d4-0c85ab2a148d"). InnerVolumeSpecName "kube-api-access-mqjkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:41:42.455992 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:42.455956 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqjkn\" (UniqueName: \"kubernetes.io/projected/7abe7663-f846-4499-b6d4-0c85ab2a148d-kube-api-access-mqjkn\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:42.455992 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:42.455985 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7abe7663-f846-4499-b6d4-0c85ab2a148d-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:43.029266 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:43.029171 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cdec0-65f74dfb55-6896h" event={"ID":"7abe7663-f846-4499-b6d4-0c85ab2a148d","Type":"ContainerDied","Data":"1b0f9da6cbd564cc2261b494f4a8454fbbafcb2eb3d29444576931902dfacc41"} Nov 02 23:41:43.029266 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:43.029197 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cdec0-65f74dfb55-6896h" Nov 02 23:41:43.029266 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:43.029235 5727 scope.go:122] "RemoveContainer" containerID="828b080ad38ee8e135f44af3bb3e58e4f44afea6f622c7d6b0611f0aaca804f9" Nov 02 23:41:43.029854 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:43.029197 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cdec0-ccc48bdc7-dkvtg" Nov 02 23:41:43.246904 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:43.246861 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4cdec0-65f74dfb55-6896h"] Nov 02 23:41:43.250754 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:43.250731 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4cdec0-65f74dfb55-6896h"] Nov 02 23:41:43.359340 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:43.359294 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4cdec0-ccc48bdc7-dkvtg"] Nov 02 23:41:43.362953 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:43.362928 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4cdec0-ccc48bdc7-dkvtg"] Nov 02 23:41:43.546968 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:43.546936 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj"] Nov 02 23:41:43.547247 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:43.547209 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" podUID="ac783482-b145-4065-b21a-0f54394f56c7" containerName="istio-proxy" containerID="cri-o://c5e64a7d7f3a3a4b3bf752c1e5e9d28a5ff24154c75640675a1500243d4dc7ad" gracePeriod=30 Nov 02 23:41:43.883657 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:43.883624 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7abe7663-f846-4499-b6d4-0c85ab2a148d" path="/var/lib/kubelet/pods/7abe7663-f846-4499-b6d4-0c85ab2a148d/volumes" Nov 02 23:41:43.884004 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:43.883991 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0b5de66-8158-4fb5-b340-d7a5b85f2766" path="/var/lib/kubelet/pods/f0b5de66-8158-4fb5-b340-d7a5b85f2766/volumes" Nov 02 23:41:48.831409 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:48.831343 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:41:48.935992 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:48.935961 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-istio-envoy\") pod \"ac783482-b145-4065-b21a-0f54394f56c7\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " Nov 02 23:41:48.936177 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:48.936016 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac783482-b145-4065-b21a-0f54394f56c7-kube-api-access-xb59s\" (UniqueName: \"kubernetes.io/projected/ac783482-b145-4065-b21a-0f54394f56c7-kube-api-access-xb59s\") pod \"ac783482-b145-4065-b21a-0f54394f56c7\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " Nov 02 23:41:48.936177 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:48.936050 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-workload-certs\") pod \"ac783482-b145-4065-b21a-0f54394f56c7\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " Nov 02 23:41:48.936177 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:48.936066 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac783482-b145-4065-b21a-0f54394f56c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac783482-b145-4065-b21a-0f54394f56c7-istiod-ca-cert\") pod \"ac783482-b145-4065-b21a-0f54394f56c7\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " Nov 02 23:41:48.936364 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:48.936188 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-istio-data\") pod \"ac783482-b145-4065-b21a-0f54394f56c7\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " Nov 02 23:41:48.936364 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:48.936212 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac783482-b145-4065-b21a-0f54394f56c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac783482-b145-4065-b21a-0f54394f56c7-istio-podinfo\") pod \"ac783482-b145-4065-b21a-0f54394f56c7\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " Nov 02 23:41:48.936364 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:48.936273 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac783482-b145-4065-b21a-0f54394f56c7-istio-token\" (UniqueName: \"kubernetes.io/projected/ac783482-b145-4065-b21a-0f54394f56c7-istio-token\") pod \"ac783482-b145-4065-b21a-0f54394f56c7\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " Nov 02 23:41:48.936364 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:48.936347 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-workload-socket\") pod \"ac783482-b145-4065-b21a-0f54394f56c7\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " Nov 02 23:41:48.936574 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:48.936353 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-workload-certs" pod "ac783482-b145-4065-b21a-0f54394f56c7" (UID: "ac783482-b145-4065-b21a-0f54394f56c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:41:48.936574 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:48.936402 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-credential-socket\") pod \"ac783482-b145-4065-b21a-0f54394f56c7\" (UID: \"ac783482-b145-4065-b21a-0f54394f56c7\") " Nov 02 23:41:48.936574 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:48.936486 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-istio-data" pod "ac783482-b145-4065-b21a-0f54394f56c7" (UID: "ac783482-b145-4065-b21a-0f54394f56c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:41:48.936574 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:48.936534 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac783482-b145-4065-b21a-0f54394f56c7-istiod-ca-cert" pod "ac783482-b145-4065-b21a-0f54394f56c7" (UID: "ac783482-b145-4065-b21a-0f54394f56c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:41:48.936780 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:48.936716 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-workload-socket" pod "ac783482-b145-4065-b21a-0f54394f56c7" (UID: "ac783482-b145-4065-b21a-0f54394f56c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:41:48.937038 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:48.936864 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-credential-socket" pod "ac783482-b145-4065-b21a-0f54394f56c7" (UID: "ac783482-b145-4065-b21a-0f54394f56c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:41:48.937479 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:48.937259 5727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-workload-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:48.937479 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:48.937284 5727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac783482-b145-4065-b21a-0f54394f56c7-istiod-ca-cert\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:48.937479 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:48.937299 5727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-istio-data\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:48.937479 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:48.937312 5727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-workload-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:48.937479 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:48.937326 5727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-credential-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:48.939161 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:48.939041 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-istio-envoy" pod "ac783482-b145-4065-b21a-0f54394f56c7" (UID: "ac783482-b145-4065-b21a-0f54394f56c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:41:48.939254 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:48.939194 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac783482-b145-4065-b21a-0f54394f56c7-istio-token" pod "ac783482-b145-4065-b21a-0f54394f56c7" (UID: "ac783482-b145-4065-b21a-0f54394f56c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:41:48.939492 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:48.939468 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac783482-b145-4065-b21a-0f54394f56c7-kube-api-access-xb59s" pod "ac783482-b145-4065-b21a-0f54394f56c7" (UID: "ac783482-b145-4065-b21a-0f54394f56c7"). InnerVolumeSpecName "kube-api-access-xb59s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:41:48.939632 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:48.939609 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac783482-b145-4065-b21a-0f54394f56c7-istio-podinfo" pod "ac783482-b145-4065-b21a-0f54394f56c7" (UID: "ac783482-b145-4065-b21a-0f54394f56c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:41:49.038948 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:49.038904 5727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac783482-b145-4065-b21a-0f54394f56c7-istio-envoy\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:49.038948 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:49.038946 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb59s\" (UniqueName: \"kubernetes.io/projected/ac783482-b145-4065-b21a-0f54394f56c7-kube-api-access-xb59s\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:49.039232 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:49.038963 5727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac783482-b145-4065-b21a-0f54394f56c7-istio-podinfo\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:49.039232 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:49.038978 5727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac783482-b145-4065-b21a-0f54394f56c7-istio-token\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:49.094789 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:49.094701 5727 generic.go:356] "Generic (PLEG): container finished" podID="ac783482-b145-4065-b21a-0f54394f56c7" containerID="c5e64a7d7f3a3a4b3bf752c1e5e9d28a5ff24154c75640675a1500243d4dc7ad" exitCode=0 Nov 02 23:41:49.094789 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:49.094762 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" Nov 02 23:41:49.094995 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:49.094787 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" event={"ID":"ac783482-b145-4065-b21a-0f54394f56c7","Type":"ContainerDied","Data":"c5e64a7d7f3a3a4b3bf752c1e5e9d28a5ff24154c75640675a1500243d4dc7ad"} Nov 02 23:41:49.094995 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:49.094890 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj" event={"ID":"ac783482-b145-4065-b21a-0f54394f56c7","Type":"ContainerDied","Data":"255ea192e78126c3ceb584caec9f6266afdbe0ca714857a3b491d807b4ff8960"} Nov 02 23:41:49.094995 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:49.094919 5727 scope.go:122] "RemoveContainer" containerID="c5e64a7d7f3a3a4b3bf752c1e5e9d28a5ff24154c75640675a1500243d4dc7ad" Nov 02 23:41:49.107647 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:49.107627 5727 scope.go:122] "RemoveContainer" containerID="c5e64a7d7f3a3a4b3bf752c1e5e9d28a5ff24154c75640675a1500243d4dc7ad" Nov 02 23:41:49.107895 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:41:49.107875 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5e64a7d7f3a3a4b3bf752c1e5e9d28a5ff24154c75640675a1500243d4dc7ad\": container with ID starting with c5e64a7d7f3a3a4b3bf752c1e5e9d28a5ff24154c75640675a1500243d4dc7ad not found: ID does not exist" containerID="c5e64a7d7f3a3a4b3bf752c1e5e9d28a5ff24154c75640675a1500243d4dc7ad" Nov 02 23:41:49.107954 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:49.107905 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5e64a7d7f3a3a4b3bf752c1e5e9d28a5ff24154c75640675a1500243d4dc7ad"} err="failed to get container status \"c5e64a7d7f3a3a4b3bf752c1e5e9d28a5ff24154c75640675a1500243d4dc7ad\": rpc error: code = NotFound desc = could not find container \"c5e64a7d7f3a3a4b3bf752c1e5e9d28a5ff24154c75640675a1500243d4dc7ad\": container with ID starting with c5e64a7d7f3a3a4b3bf752c1e5e9d28a5ff24154c75640675a1500243d4dc7ad not found: ID does not exist" Nov 02 23:41:49.262965 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:49.262936 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj"] Nov 02 23:41:49.268394 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:49.268360 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-zm9dj"] Nov 02 23:41:49.883711 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:41:49.883675 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac783482-b145-4065-b21a-0f54394f56c7" path="/var/lib/kubelet/pods/ac783482-b145-4065-b21a-0f54394f56c7/volumes" Nov 02 23:41:57.846259 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:41:57.846220 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:42:27.845852 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:42:27.845817 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:42:35.036478 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:42:35.036443 5727 scope.go:122] "RemoveContainer" containerID="d13dd821c2a4626b5ba67c049704e1fc36804e3c3841bb4ceb2167df350ff3ac" Nov 02 23:42:57.847416 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:42:57.847376 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:43:27.847068 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:43:27.847031 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:43:34.675917 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:43:34.675887 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 02 23:43:34.695264 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:43:34.695242 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 23:43:34.747633 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:43:34.747605 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 02 23:43:34.764331 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:43:34.764302 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 23:43:57.893151 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:43:57.892242 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:44:27.846038 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:44:27.845995 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:44:57.844800 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:44:57.844740 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:45:00.201497 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:00.201456 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx"] Nov 02 23:45:00.202206 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:00.202191 5727 cpu_manager.go:414] "containerMap: removing container" podUID="f0b5de66-8158-4fb5-b340-d7a5b85f2766" containerName="payload-processing" Nov 02 23:45:00.202275 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:00.202207 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0b5de66-8158-4fb5-b340-d7a5b85f2766" containerName="payload-processing" Nov 02 23:45:00.202275 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:00.202255 5727 cpu_manager.go:414] "containerMap: removing container" podUID="7abe7663-f846-4499-b6d4-0c85ab2a148d" containerName="payload-pre-processing" Nov 02 23:45:00.202275 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:00.202261 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7abe7663-f846-4499-b6d4-0c85ab2a148d" containerName="payload-pre-processing" Nov 02 23:45:00.202385 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:00.202287 5727 cpu_manager.go:414] "containerMap: removing container" podUID="4ec013db-b396-4d4e-8d00-5cf3f957b9c8" containerName="payload-pre-processing" Nov 02 23:45:00.202385 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:00.202292 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ec013db-b396-4d4e-8d00-5cf3f957b9c8" containerName="payload-pre-processing" Nov 02 23:45:00.202385 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:00.202300 5727 cpu_manager.go:414] "containerMap: removing container" podUID="bf35636b-a796-4dfc-83af-9f052a649ae3" containerName="istio-proxy" Nov 02 23:45:00.202385 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:00.202304 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf35636b-a796-4dfc-83af-9f052a649ae3" containerName="istio-proxy" Nov 02 23:45:00.202385 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:00.202312 5727 cpu_manager.go:414] "containerMap: removing container" podUID="ac783482-b145-4065-b21a-0f54394f56c7" containerName="istio-proxy" Nov 02 23:45:00.202385 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:00.202317 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac783482-b145-4065-b21a-0f54394f56c7" containerName="istio-proxy" Nov 02 23:45:00.202570 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:00.202411 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="7abe7663-f846-4499-b6d4-0c85ab2a148d" containerName="payload-pre-processing" Nov 02 23:45:00.202570 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:00.202423 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ec013db-b396-4d4e-8d00-5cf3f957b9c8" containerName="payload-pre-processing" Nov 02 23:45:00.202570 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:00.202430 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf35636b-a796-4dfc-83af-9f052a649ae3" containerName="istio-proxy" Nov 02 23:45:00.202570 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:00.202436 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac783482-b145-4065-b21a-0f54394f56c7" containerName="istio-proxy" Nov 02 23:45:00.202570 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:00.202442 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0b5de66-8158-4fb5-b340-d7a5b85f2766" containerName="payload-processing" Nov 02 23:45:00.206104 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:00.206089 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx" Nov 02 23:45:00.252064 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:00.252033 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t2lf\" (UniqueName: \"kubernetes.io/projected/2bad7bf4-87d2-4e0c-b18e-adb154740d49-kube-api-access-7t2lf\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx\" (UID: \"2bad7bf4-87d2-4e0c-b18e-adb154740d49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx" Nov 02 23:45:00.252225 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:00.252089 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bad7bf4-87d2-4e0c-b18e-adb154740d49-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx\" (UID: \"2bad7bf4-87d2-4e0c-b18e-adb154740d49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx" Nov 02 23:45:00.318216 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:00.318181 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx"] Nov 02 23:45:00.343562 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:00.343519 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:45:00.353637 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:00.353605 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t2lf\" (UniqueName: \"kubernetes.io/projected/2bad7bf4-87d2-4e0c-b18e-adb154740d49-kube-api-access-7t2lf\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx\" (UID: \"2bad7bf4-87d2-4e0c-b18e-adb154740d49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx" Nov 02 23:45:00.353805 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:00.353671 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bad7bf4-87d2-4e0c-b18e-adb154740d49-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx\" (UID: \"2bad7bf4-87d2-4e0c-b18e-adb154740d49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx" Nov 02 23:45:00.354550 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:00.354522 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bad7bf4-87d2-4e0c-b18e-adb154740d49-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx\" (UID: \"2bad7bf4-87d2-4e0c-b18e-adb154740d49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx" Nov 02 23:45:00.414326 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:00.414292 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t2lf\" (UniqueName: \"kubernetes.io/projected/2bad7bf4-87d2-4e0c-b18e-adb154740d49-kube-api-access-7t2lf\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx\" (UID: \"2bad7bf4-87d2-4e0c-b18e-adb154740d49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx" Nov 02 23:45:00.520026 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:00.519952 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx" Nov 02 23:45:00.903372 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:00.903343 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx"] Nov 02 23:45:00.903984 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:45:00.903958 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bad7bf4_87d2_4e0c_b18e_adb154740d49.slice/crio-4efc350c831b7628f2c87964e1f4e9afb1034a7be0b4fe4d48e8a4c68b9fcb4c WatchSource:0}: Error finding container 4efc350c831b7628f2c87964e1f4e9afb1034a7be0b4fe4d48e8a4c68b9fcb4c: Status 404 returned error can't find the container with id 4efc350c831b7628f2c87964e1f4e9afb1034a7be0b4fe4d48e8a4c68b9fcb4c Nov 02 23:45:01.638709 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:01.638669 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx" event={"ID":"2bad7bf4-87d2-4e0c-b18e-adb154740d49","Type":"ContainerStarted","Data":"e8a46f1080d75230303756b8a707de68c432fd3ddc73717b2c69cfd349ed1a20"} Nov 02 23:45:01.639084 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:01.638734 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx" event={"ID":"2bad7bf4-87d2-4e0c-b18e-adb154740d49","Type":"ContainerStarted","Data":"4efc350c831b7628f2c87964e1f4e9afb1034a7be0b4fe4d48e8a4c68b9fcb4c"} Nov 02 23:45:01.756926 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:01.756561 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx" podStartSLOduration=1.756545354 podStartE2EDuration="1.756545354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:01.756331902 +0000 UTC m=+3387.994829564" watchObservedRunningTime="2026-11-02 23:45:01.756545354 +0000 UTC m=+3387.995043019" Nov 02 23:45:02.647551 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:02.647513 5727 generic.go:356] "Generic (PLEG): container finished" podID="2bad7bf4-87d2-4e0c-b18e-adb154740d49" containerID="e8a46f1080d75230303756b8a707de68c432fd3ddc73717b2c69cfd349ed1a20" exitCode=60 Nov 02 23:45:02.648016 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:02.647603 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx" event={"ID":"2bad7bf4-87d2-4e0c-b18e-adb154740d49","Type":"ContainerDied","Data":"e8a46f1080d75230303756b8a707de68c432fd3ddc73717b2c69cfd349ed1a20"} Nov 02 23:45:02.648016 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:02.647816 5727 scope.go:122] "RemoveContainer" containerID="e8a46f1080d75230303756b8a707de68c432fd3ddc73717b2c69cfd349ed1a20" Nov 02 23:45:03.659183 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:03.659121 5727 generic.go:356] "Generic (PLEG): container finished" podID="2bad7bf4-87d2-4e0c-b18e-adb154740d49" containerID="638e4e371de0f1b1be0588b0e906ed8de8fa9a99dcf37a38b183592162d4f3a4" exitCode=60 Nov 02 23:45:03.659686 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:03.659223 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx" event={"ID":"2bad7bf4-87d2-4e0c-b18e-adb154740d49","Type":"ContainerDied","Data":"638e4e371de0f1b1be0588b0e906ed8de8fa9a99dcf37a38b183592162d4f3a4"} Nov 02 23:45:03.659686 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:03.659318 5727 scope.go:122] "RemoveContainer" containerID="e8a46f1080d75230303756b8a707de68c432fd3ddc73717b2c69cfd349ed1a20" Nov 02 23:45:03.659686 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:03.659653 5727 scope.go:122] "RemoveContainer" containerID="638e4e371de0f1b1be0588b0e906ed8de8fa9a99dcf37a38b183592162d4f3a4" Nov 02 23:45:03.660037 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:45:03.660015 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx_odh-ai-gateway-infra(2bad7bf4-87d2-4e0c-b18e-adb154740d49)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx" podUID="2bad7bf4-87d2-4e0c-b18e-adb154740d49" Nov 02 23:45:04.672147 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:04.672102 5727 scope.go:122] "RemoveContainer" containerID="638e4e371de0f1b1be0588b0e906ed8de8fa9a99dcf37a38b183592162d4f3a4" Nov 02 23:45:04.672632 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:45:04.672496 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx_odh-ai-gateway-infra(2bad7bf4-87d2-4e0c-b18e-adb154740d49)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx" podUID="2bad7bf4-87d2-4e0c-b18e-adb154740d49" Nov 02 23:45:17.876308 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:17.876277 5727 scope.go:122] "RemoveContainer" containerID="638e4e371de0f1b1be0588b0e906ed8de8fa9a99dcf37a38b183592162d4f3a4" Nov 02 23:45:18.793459 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:18.793426 5727 generic.go:356] "Generic (PLEG): container finished" podID="2bad7bf4-87d2-4e0c-b18e-adb154740d49" containerID="b25a9f4dec52da707e8d304c6b7fd1258dc4d8fcfa0935c85fb98605b77f0a54" exitCode=60 Nov 02 23:45:18.793578 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:18.793498 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx" event={"ID":"2bad7bf4-87d2-4e0c-b18e-adb154740d49","Type":"ContainerDied","Data":"b25a9f4dec52da707e8d304c6b7fd1258dc4d8fcfa0935c85fb98605b77f0a54"} Nov 02 23:45:18.793623 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:18.793595 5727 scope.go:122] "RemoveContainer" containerID="638e4e371de0f1b1be0588b0e906ed8de8fa9a99dcf37a38b183592162d4f3a4" Nov 02 23:45:18.793963 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:18.793946 5727 scope.go:122] "RemoveContainer" containerID="b25a9f4dec52da707e8d304c6b7fd1258dc4d8fcfa0935c85fb98605b77f0a54" Nov 02 23:45:18.794320 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:45:18.794292 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx_odh-ai-gateway-infra(2bad7bf4-87d2-4e0c-b18e-adb154740d49)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx" podUID="2bad7bf4-87d2-4e0c-b18e-adb154740d49" Nov 02 23:45:19.919222 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:19.919187 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx"] Nov 02 23:45:20.079666 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:20.079640 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx" Nov 02 23:45:20.206419 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:20.206337 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bad7bf4-87d2-4e0c-b18e-adb154740d49-kube-api-access-7t2lf\" (UniqueName: \"kubernetes.io/projected/2bad7bf4-87d2-4e0c-b18e-adb154740d49-kube-api-access-7t2lf\") pod \"2bad7bf4-87d2-4e0c-b18e-adb154740d49\" (UID: \"2bad7bf4-87d2-4e0c-b18e-adb154740d49\") " Nov 02 23:45:20.206419 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:20.206380 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bad7bf4-87d2-4e0c-b18e-adb154740d49-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bad7bf4-87d2-4e0c-b18e-adb154740d49-maas-api-service-ca\") pod \"2bad7bf4-87d2-4e0c-b18e-adb154740d49\" (UID: \"2bad7bf4-87d2-4e0c-b18e-adb154740d49\") " Nov 02 23:45:20.206791 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:20.206759 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bad7bf4-87d2-4e0c-b18e-adb154740d49-maas-api-service-ca" pod "2bad7bf4-87d2-4e0c-b18e-adb154740d49" (UID: "2bad7bf4-87d2-4e0c-b18e-adb154740d49"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:45:20.206925 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:20.206849 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bad7bf4-87d2-4e0c-b18e-adb154740d49-maas-api-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:20.208694 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:20.208674 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bad7bf4-87d2-4e0c-b18e-adb154740d49-kube-api-access-7t2lf" pod "2bad7bf4-87d2-4e0c-b18e-adb154740d49" (UID: "2bad7bf4-87d2-4e0c-b18e-adb154740d49"). InnerVolumeSpecName "kube-api-access-7t2lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:20.308992 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:20.308959 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t2lf\" (UniqueName: \"kubernetes.io/projected/2bad7bf4-87d2-4e0c-b18e-adb154740d49-kube-api-access-7t2lf\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:20.813334 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:20.813300 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx" Nov 02 23:45:20.813515 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:20.813350 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx" event={"ID":"2bad7bf4-87d2-4e0c-b18e-adb154740d49","Type":"ContainerDied","Data":"4efc350c831b7628f2c87964e1f4e9afb1034a7be0b4fe4d48e8a4c68b9fcb4c"} Nov 02 23:45:20.813515 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:20.813405 5727 scope.go:122] "RemoveContainer" containerID="b25a9f4dec52da707e8d304c6b7fd1258dc4d8fcfa0935c85fb98605b77f0a54" Nov 02 23:45:20.979460 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:20.979407 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx"] Nov 02 23:45:20.984650 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:20.984622 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894385-wrtrx"] Nov 02 23:45:21.885351 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:21.885318 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bad7bf4-87d2-4e0c-b18e-adb154740d49" path="/var/lib/kubelet/pods/2bad7bf4-87d2-4e0c-b18e-adb154740d49/volumes" Nov 02 23:45:27.853338 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:45:27.853291 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:45:43.404029 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:43.403939 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z"] Nov 02 23:45:57.846730 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:45:57.846685 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:45:58.425910 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:58.425864 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" podUID="0fcc517d-e223-4597-99c4-9deaaf163659" containerName="main" containerID="cri-o://d8262f6deebc2b694908dd7a7039da01c8ca503c74681186ba4debb59892ae5a" gracePeriod=45 Nov 02 23:45:58.442806 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:58.442771 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" podUID="0fcc517d-e223-4597-99c4-9deaaf163659" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": dial tcp 10.132.0.160:8000: connect: connection refused" Nov 02 23:45:58.721534 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:58.721504 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" Nov 02 23:45:58.840181 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:58.839164 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fcc517d-e223-4597-99c4-9deaaf163659-tls-certs\" (UniqueName: \"kubernetes.io/secret/0fcc517d-e223-4597-99c4-9deaaf163659-tls-certs\") pod \"0fcc517d-e223-4597-99c4-9deaaf163659\" (UID: \"0fcc517d-e223-4597-99c4-9deaaf163659\") " Nov 02 23:45:58.840181 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:58.839260 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fcc517d-e223-4597-99c4-9deaaf163659-home\" (UniqueName: \"kubernetes.io/empty-dir/0fcc517d-e223-4597-99c4-9deaaf163659-home\") pod \"0fcc517d-e223-4597-99c4-9deaaf163659\" (UID: \"0fcc517d-e223-4597-99c4-9deaaf163659\") " Nov 02 23:45:58.840181 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:58.839295 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fcc517d-e223-4597-99c4-9deaaf163659-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fcc517d-e223-4597-99c4-9deaaf163659-dshm\") pod \"0fcc517d-e223-4597-99c4-9deaaf163659\" (UID: \"0fcc517d-e223-4597-99c4-9deaaf163659\") " Nov 02 23:45:58.840181 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:58.839445 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fcc517d-e223-4597-99c4-9deaaf163659-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fcc517d-e223-4597-99c4-9deaaf163659-model-cache\") pod \"0fcc517d-e223-4597-99c4-9deaaf163659\" (UID: \"0fcc517d-e223-4597-99c4-9deaaf163659\") " Nov 02 23:45:58.840181 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:58.839490 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fcc517d-e223-4597-99c4-9deaaf163659-kube-api-access-ppfmk\" (UniqueName: \"kubernetes.io/projected/0fcc517d-e223-4597-99c4-9deaaf163659-kube-api-access-ppfmk\") pod \"0fcc517d-e223-4597-99c4-9deaaf163659\" (UID: \"0fcc517d-e223-4597-99c4-9deaaf163659\") " Nov 02 23:45:58.840181 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:58.839652 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fcc517d-e223-4597-99c4-9deaaf163659-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fcc517d-e223-4597-99c4-9deaaf163659-tmp-dir\") pod \"0fcc517d-e223-4597-99c4-9deaaf163659\" (UID: \"0fcc517d-e223-4597-99c4-9deaaf163659\") " Nov 02 23:45:58.841100 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:58.840350 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fcc517d-e223-4597-99c4-9deaaf163659-tmp-dir" pod "0fcc517d-e223-4597-99c4-9deaaf163659" (UID: "0fcc517d-e223-4597-99c4-9deaaf163659"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:45:58.841100 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:58.841069 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fcc517d-e223-4597-99c4-9deaaf163659-home" pod "0fcc517d-e223-4597-99c4-9deaaf163659" (UID: "0fcc517d-e223-4597-99c4-9deaaf163659"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:45:58.841861 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:58.841822 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fcc517d-e223-4597-99c4-9deaaf163659-model-cache" pod "0fcc517d-e223-4597-99c4-9deaaf163659" (UID: "0fcc517d-e223-4597-99c4-9deaaf163659"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:45:58.847516 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:58.845590 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fcc517d-e223-4597-99c4-9deaaf163659-dshm" pod "0fcc517d-e223-4597-99c4-9deaaf163659" (UID: "0fcc517d-e223-4597-99c4-9deaaf163659"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:45:58.847516 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:58.846635 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fcc517d-e223-4597-99c4-9deaaf163659-tls-certs" pod "0fcc517d-e223-4597-99c4-9deaaf163659" (UID: "0fcc517d-e223-4597-99c4-9deaaf163659"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:45:58.853346 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:58.853295 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fcc517d-e223-4597-99c4-9deaaf163659-kube-api-access-ppfmk" pod "0fcc517d-e223-4597-99c4-9deaaf163659" (UID: "0fcc517d-e223-4597-99c4-9deaaf163659"). InnerVolumeSpecName "kube-api-access-ppfmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:58.941695 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:58.941604 5727 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fcc517d-e223-4597-99c4-9deaaf163659-home\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:58.941695 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:58.941636 5727 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fcc517d-e223-4597-99c4-9deaaf163659-dshm\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:58.941695 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:58.941649 5727 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fcc517d-e223-4597-99c4-9deaaf163659-model-cache\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:58.941695 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:58.941661 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppfmk\" (UniqueName: \"kubernetes.io/projected/0fcc517d-e223-4597-99c4-9deaaf163659-kube-api-access-ppfmk\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:58.941695 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:58.941676 5727 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fcc517d-e223-4597-99c4-9deaaf163659-tmp-dir\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:58.941695 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:58.941690 5727 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fcc517d-e223-4597-99c4-9deaaf163659-tls-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:59.117804 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:59.117767 5727 generic.go:356] "Generic (PLEG): container finished" podID="0fcc517d-e223-4597-99c4-9deaaf163659" containerID="d8262f6deebc2b694908dd7a7039da01c8ca503c74681186ba4debb59892ae5a" exitCode=0 Nov 02 23:45:59.117993 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:59.117810 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" Nov 02 23:45:59.117993 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:59.117848 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" event={"ID":"0fcc517d-e223-4597-99c4-9deaaf163659","Type":"ContainerDied","Data":"d8262f6deebc2b694908dd7a7039da01c8ca503c74681186ba4debb59892ae5a"} Nov 02 23:45:59.117993 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:59.117891 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z" event={"ID":"0fcc517d-e223-4597-99c4-9deaaf163659","Type":"ContainerDied","Data":"f06604e79bf87cd4f88949a40e068d15a9a0184293486552a5f94e0d88e30476"} Nov 02 23:45:59.117993 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:59.117908 5727 scope.go:122] "RemoveContainer" containerID="d8262f6deebc2b694908dd7a7039da01c8ca503c74681186ba4debb59892ae5a" Nov 02 23:45:59.129945 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:59.129922 5727 scope.go:122] "RemoveContainer" containerID="d8262f6deebc2b694908dd7a7039da01c8ca503c74681186ba4debb59892ae5a" Nov 02 23:45:59.130289 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:45:59.130269 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8262f6deebc2b694908dd7a7039da01c8ca503c74681186ba4debb59892ae5a\": container with ID starting with d8262f6deebc2b694908dd7a7039da01c8ca503c74681186ba4debb59892ae5a not found: ID does not exist" containerID="d8262f6deebc2b694908dd7a7039da01c8ca503c74681186ba4debb59892ae5a" Nov 02 23:45:59.130343 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:59.130299 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8262f6deebc2b694908dd7a7039da01c8ca503c74681186ba4debb59892ae5a"} err="failed to get container status \"d8262f6deebc2b694908dd7a7039da01c8ca503c74681186ba4debb59892ae5a\": rpc error: code = NotFound desc = could not find container \"d8262f6deebc2b694908dd7a7039da01c8ca503c74681186ba4debb59892ae5a\": container with ID starting with d8262f6deebc2b694908dd7a7039da01c8ca503c74681186ba4debb59892ae5a not found: ID does not exist" Nov 02 23:45:59.282457 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:59.282360 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z"] Nov 02 23:45:59.334269 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:59.334233 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-440c60/auth-test-model-440c60-kserve-5579dbbdbf-2mh9z"] Nov 02 23:45:59.905117 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:45:59.905068 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fcc517d-e223-4597-99c4-9deaaf163659" path="/var/lib/kubelet/pods/0fcc517d-e223-4597-99c4-9deaaf163659/volumes" Nov 02 23:46:14.292482 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:14.292449 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz"] Nov 02 23:46:27.846992 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:46:27.846948 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:46:29.314209 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:29.314168 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz" podUID="aaa5326f-dbfd-48cd-bf98-830703edb686" containerName="main" containerID="cri-o://e0863f5cf28656e4005ad8e5086c41aa6d85e2c7d8879252af55a02870e859c2" gracePeriod=45 Nov 02 23:46:29.433153 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:29.433106 5727 generic.go:356] "Generic (PLEG): container finished" podID="aaa5326f-dbfd-48cd-bf98-830703edb686" containerID="e0863f5cf28656e4005ad8e5086c41aa6d85e2c7d8879252af55a02870e859c2" exitCode=0 Nov 02 23:46:29.433292 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:29.433160 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz" event={"ID":"aaa5326f-dbfd-48cd-bf98-830703edb686","Type":"ContainerDied","Data":"e0863f5cf28656e4005ad8e5086c41aa6d85e2c7d8879252af55a02870e859c2"} Nov 02 23:46:29.609483 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:29.609454 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz" Nov 02 23:46:29.648660 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:29.648628 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaa5326f-dbfd-48cd-bf98-830703edb686-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaa5326f-dbfd-48cd-bf98-830703edb686-model-cache\") pod \"aaa5326f-dbfd-48cd-bf98-830703edb686\" (UID: \"aaa5326f-dbfd-48cd-bf98-830703edb686\") " Nov 02 23:46:29.648795 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:29.648743 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaa5326f-dbfd-48cd-bf98-830703edb686-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaa5326f-dbfd-48cd-bf98-830703edb686-tmp-dir\") pod \"aaa5326f-dbfd-48cd-bf98-830703edb686\" (UID: \"aaa5326f-dbfd-48cd-bf98-830703edb686\") " Nov 02 23:46:29.648860 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:29.648820 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaa5326f-dbfd-48cd-bf98-830703edb686-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aaa5326f-dbfd-48cd-bf98-830703edb686-dshm\") pod \"aaa5326f-dbfd-48cd-bf98-830703edb686\" (UID: \"aaa5326f-dbfd-48cd-bf98-830703edb686\") " Nov 02 23:46:29.648918 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:29.648855 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaa5326f-dbfd-48cd-bf98-830703edb686-model-cache" pod "aaa5326f-dbfd-48cd-bf98-830703edb686" (UID: "aaa5326f-dbfd-48cd-bf98-830703edb686"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:46:29.648918 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:29.648874 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaa5326f-dbfd-48cd-bf98-830703edb686-home\" (UniqueName: \"kubernetes.io/empty-dir/aaa5326f-dbfd-48cd-bf98-830703edb686-home\") pod \"aaa5326f-dbfd-48cd-bf98-830703edb686\" (UID: \"aaa5326f-dbfd-48cd-bf98-830703edb686\") " Nov 02 23:46:29.649030 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:29.648918 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaa5326f-dbfd-48cd-bf98-830703edb686-tls-certs\" (UniqueName: \"kubernetes.io/secret/aaa5326f-dbfd-48cd-bf98-830703edb686-tls-certs\") pod \"aaa5326f-dbfd-48cd-bf98-830703edb686\" (UID: \"aaa5326f-dbfd-48cd-bf98-830703edb686\") " Nov 02 23:46:29.649030 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:29.648990 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaa5326f-dbfd-48cd-bf98-830703edb686-kube-api-access-fcjdq\" (UniqueName: \"kubernetes.io/projected/aaa5326f-dbfd-48cd-bf98-830703edb686-kube-api-access-fcjdq\") pod \"aaa5326f-dbfd-48cd-bf98-830703edb686\" (UID: \"aaa5326f-dbfd-48cd-bf98-830703edb686\") " Nov 02 23:46:29.649030 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:29.649016 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaa5326f-dbfd-48cd-bf98-830703edb686-tmp-dir" pod "aaa5326f-dbfd-48cd-bf98-830703edb686" (UID: "aaa5326f-dbfd-48cd-bf98-830703edb686"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:46:29.649225 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:29.649051 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaa5326f-dbfd-48cd-bf98-830703edb686-home" pod "aaa5326f-dbfd-48cd-bf98-830703edb686" (UID: "aaa5326f-dbfd-48cd-bf98-830703edb686"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:46:29.649521 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:29.649489 5727 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaa5326f-dbfd-48cd-bf98-830703edb686-tmp-dir\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:29.649594 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:29.649528 5727 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aaa5326f-dbfd-48cd-bf98-830703edb686-home\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:29.649594 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:29.649541 5727 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaa5326f-dbfd-48cd-bf98-830703edb686-model-cache\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:29.651177 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:29.651147 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaa5326f-dbfd-48cd-bf98-830703edb686-dshm" pod "aaa5326f-dbfd-48cd-bf98-830703edb686" (UID: "aaa5326f-dbfd-48cd-bf98-830703edb686"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:46:29.651261 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:29.651222 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaa5326f-dbfd-48cd-bf98-830703edb686-tls-certs" pod "aaa5326f-dbfd-48cd-bf98-830703edb686" (UID: "aaa5326f-dbfd-48cd-bf98-830703edb686"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:46:29.651261 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:29.651249 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaa5326f-dbfd-48cd-bf98-830703edb686-kube-api-access-fcjdq" pod "aaa5326f-dbfd-48cd-bf98-830703edb686" (UID: "aaa5326f-dbfd-48cd-bf98-830703edb686"). InnerVolumeSpecName "kube-api-access-fcjdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:46:29.750621 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:29.750584 5727 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aaa5326f-dbfd-48cd-bf98-830703edb686-dshm\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:29.750621 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:29.750622 5727 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaa5326f-dbfd-48cd-bf98-830703edb686-tls-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:29.750805 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:29.750638 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcjdq\" (UniqueName: \"kubernetes.io/projected/aaa5326f-dbfd-48cd-bf98-830703edb686-kube-api-access-fcjdq\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:30.442275 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:30.442245 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz" Nov 02 23:46:30.442709 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:30.442275 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz" event={"ID":"aaa5326f-dbfd-48cd-bf98-830703edb686","Type":"ContainerDied","Data":"677118009ab076520579e980235f2b88d7fb12b99c676c7ab0e1260402d22c82"} Nov 02 23:46:30.442709 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:30.442325 5727 scope.go:122] "RemoveContainer" containerID="e0863f5cf28656e4005ad8e5086c41aa6d85e2c7d8879252af55a02870e859c2" Nov 02 23:46:30.606947 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:30.606917 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz"] Nov 02 23:46:30.612055 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:30.612032 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-915ccb/auth-test-model-915ccb-kserve-86f59f45d5-6crmz"] Nov 02 23:46:31.885207 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:31.885170 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaa5326f-dbfd-48cd-bf98-830703edb686" path="/var/lib/kubelet/pods/aaa5326f-dbfd-48cd-bf98-830703edb686/volumes" Nov 02 23:46:43.199587 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:43.199545 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0f49ab-557c48887f-mt5fg"] Nov 02 23:46:43.200535 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:43.200511 5727 cpu_manager.go:414] "containerMap: removing container" podUID="aaa5326f-dbfd-48cd-bf98-830703edb686" containerName="main" Nov 02 23:46:43.200652 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:43.200537 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaa5326f-dbfd-48cd-bf98-830703edb686" containerName="main" Nov 02 23:46:43.200652 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:43.200564 5727 cpu_manager.go:414] "containerMap: removing container" podUID="2bad7bf4-87d2-4e0c-b18e-adb154740d49" containerName="cleanup" Nov 02 23:46:43.200652 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:43.200572 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bad7bf4-87d2-4e0c-b18e-adb154740d49" containerName="cleanup" Nov 02 23:46:43.200652 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:43.200633 5727 cpu_manager.go:414] "containerMap: removing container" podUID="2bad7bf4-87d2-4e0c-b18e-adb154740d49" containerName="cleanup" Nov 02 23:46:43.200652 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:43.200640 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bad7bf4-87d2-4e0c-b18e-adb154740d49" containerName="cleanup" Nov 02 23:46:43.200652 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:43.200652 5727 cpu_manager.go:414] "containerMap: removing container" podUID="0fcc517d-e223-4597-99c4-9deaaf163659" containerName="main" Nov 02 23:46:43.200949 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:43.200659 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fcc517d-e223-4597-99c4-9deaaf163659" containerName="main" Nov 02 23:46:43.200949 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:43.200811 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bad7bf4-87d2-4e0c-b18e-adb154740d49" containerName="cleanup" Nov 02 23:46:43.200949 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:43.200832 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bad7bf4-87d2-4e0c-b18e-adb154740d49" containerName="cleanup" Nov 02 23:46:43.200949 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:43.200839 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bad7bf4-87d2-4e0c-b18e-adb154740d49" containerName="cleanup" Nov 02 23:46:43.200949 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:43.200848 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fcc517d-e223-4597-99c4-9deaaf163659" containerName="main" Nov 02 23:46:43.200949 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:43.200861 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaa5326f-dbfd-48cd-bf98-830703edb686" containerName="main" Nov 02 23:46:43.205376 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:43.205346 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f49ab-557c48887f-mt5fg" Nov 02 23:46:43.256843 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:43.256812 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0f49ab-557c48887f-mt5fg"] Nov 02 23:46:43.302962 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:43.302530 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-0f49ab\"" Nov 02 23:46:43.302962 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:43.302699 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-0f49ab-dockercfg-2q98d\"" Nov 02 23:46:43.331478 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:43.331445 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dps79\" (UniqueName: \"kubernetes.io/projected/453c1e89-68c1-4883-ba1b-d2a19c6b2dfb-kube-api-access-dps79\") pod \"payload-processing-e2e-extmod-a-0f49ab-557c48887f-mt5fg\" (UID: \"453c1e89-68c1-4883-ba1b-d2a19c6b2dfb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0f49ab-557c48887f-mt5fg" Nov 02 23:46:43.331737 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:43.331714 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/453c1e89-68c1-4883-ba1b-d2a19c6b2dfb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0f49ab-557c48887f-mt5fg\" (UID: \"453c1e89-68c1-4883-ba1b-d2a19c6b2dfb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0f49ab-557c48887f-mt5fg" Nov 02 23:46:43.432653 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:43.432619 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/453c1e89-68c1-4883-ba1b-d2a19c6b2dfb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0f49ab-557c48887f-mt5fg\" (UID: \"453c1e89-68c1-4883-ba1b-d2a19c6b2dfb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0f49ab-557c48887f-mt5fg" Nov 02 23:46:43.432833 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:43.432672 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dps79\" (UniqueName: \"kubernetes.io/projected/453c1e89-68c1-4883-ba1b-d2a19c6b2dfb-kube-api-access-dps79\") pod \"payload-processing-e2e-extmod-a-0f49ab-557c48887f-mt5fg\" (UID: \"453c1e89-68c1-4883-ba1b-d2a19c6b2dfb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0f49ab-557c48887f-mt5fg" Nov 02 23:46:43.433419 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:43.433395 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/453c1e89-68c1-4883-ba1b-d2a19c6b2dfb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0f49ab-557c48887f-mt5fg\" (UID: \"453c1e89-68c1-4883-ba1b-d2a19c6b2dfb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0f49ab-557c48887f-mt5fg" Nov 02 23:46:43.493527 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:43.493449 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dps79\" (UniqueName: \"kubernetes.io/projected/453c1e89-68c1-4883-ba1b-d2a19c6b2dfb-kube-api-access-dps79\") pod \"payload-processing-e2e-extmod-a-0f49ab-557c48887f-mt5fg\" (UID: \"453c1e89-68c1-4883-ba1b-d2a19c6b2dfb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0f49ab-557c48887f-mt5fg" Nov 02 23:46:43.523495 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:43.523464 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f49ab-557c48887f-mt5fg" Nov 02 23:46:43.908043 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:43.908011 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0f49ab-557c48887f-mt5fg"] Nov 02 23:46:43.909166 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:46:43.909123 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod453c1e89_68c1_4883_ba1b_d2a19c6b2dfb.slice/crio-2edc951fed7e6c5f8b1b09b9f95c234d3ba6a57164f16a151574b7ccb074f487 WatchSource:0}: Error finding container 2edc951fed7e6c5f8b1b09b9f95c234d3ba6a57164f16a151574b7ccb074f487: Status 404 returned error can't find the container with id 2edc951fed7e6c5f8b1b09b9f95c234d3ba6a57164f16a151574b7ccb074f487 Nov 02 23:46:44.562907 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:44.562863 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f49ab-557c48887f-mt5fg" event={"ID":"453c1e89-68c1-4883-ba1b-d2a19c6b2dfb","Type":"ContainerStarted","Data":"06f07e95b4569b4252c03167cf012a9e817c4d5658a9d3c1567152fb3ea8bc46"} Nov 02 23:46:44.563410 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:44.562936 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f49ab-557c48887f-mt5fg" Nov 02 23:46:44.563410 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:44.562960 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f49ab-557c48887f-mt5fg" event={"ID":"453c1e89-68c1-4883-ba1b-d2a19c6b2dfb","Type":"ContainerStarted","Data":"2edc951fed7e6c5f8b1b09b9f95c234d3ba6a57164f16a151574b7ccb074f487"} Nov 02 23:46:44.679716 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:44.679661 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f49ab-557c48887f-mt5fg" podStartSLOduration=1.679647236 podStartE2EDuration="1.679647236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:46:44.676195106 +0000 UTC m=+3490.914692769" watchObservedRunningTime="2026-11-02 23:46:44.679647236 +0000 UTC m=+3490.918144897" Nov 02 23:46:45.865538 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:45.865499 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v"] Nov 02 23:46:45.866018 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:45.865716 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" podUID="575d202b-f514-4312-8dc7-dfad51c6ca19" containerName="istio-proxy" containerID="cri-o://983d66f724bc1f6dac51c13a78a6ac56816866652d92394fdee53a768d587f1b" gracePeriod=30 Nov 02 23:46:48.516887 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:48.516842 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7"] Nov 02 23:46:48.517428 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:48.517173 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" podUID="e5415246-c0df-4f8d-a382-34a674b1f21f" containerName="istio-proxy" containerID="cri-o://477dc86b29f9b9ea4eea1c7c2b1ad30cd7391097ae1eb7d600fbfda318afd05e" gracePeriod=30 Nov 02 23:46:50.937187 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:50.937118 5727 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:46:50.937641 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:50.937219 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" podUID="e5415246-c0df-4f8d-a382-34a674b1f21f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:46:51.187017 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.186983 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:46:51.331636 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.331597 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-credential-socket\") pod \"575d202b-f514-4312-8dc7-dfad51c6ca19\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " Nov 02 23:46:51.331792 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.331708 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/575d202b-f514-4312-8dc7-dfad51c6ca19-kube-api-access-qddss\" (UniqueName: \"kubernetes.io/projected/575d202b-f514-4312-8dc7-dfad51c6ca19-kube-api-access-qddss\") pod \"575d202b-f514-4312-8dc7-dfad51c6ca19\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " Nov 02 23:46:51.331792 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.331778 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/575d202b-f514-4312-8dc7-dfad51c6ca19-istio-token\" (UniqueName: \"kubernetes.io/projected/575d202b-f514-4312-8dc7-dfad51c6ca19-istio-token\") pod \"575d202b-f514-4312-8dc7-dfad51c6ca19\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " Nov 02 23:46:51.331931 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.331874 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-credential-socket" pod "575d202b-f514-4312-8dc7-dfad51c6ca19" (UID: "575d202b-f514-4312-8dc7-dfad51c6ca19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:46:51.331931 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.331891 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/575d202b-f514-4312-8dc7-dfad51c6ca19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/575d202b-f514-4312-8dc7-dfad51c6ca19-istiod-ca-cert\") pod \"575d202b-f514-4312-8dc7-dfad51c6ca19\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " Nov 02 23:46:51.332045 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.331943 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-istio-envoy\") pod \"575d202b-f514-4312-8dc7-dfad51c6ca19\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " Nov 02 23:46:51.332045 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.331982 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-workload-certs\") pod \"575d202b-f514-4312-8dc7-dfad51c6ca19\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " Nov 02 23:46:51.332045 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.332013 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/575d202b-f514-4312-8dc7-dfad51c6ca19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/575d202b-f514-4312-8dc7-dfad51c6ca19-istio-podinfo\") pod \"575d202b-f514-4312-8dc7-dfad51c6ca19\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " Nov 02 23:46:51.332045 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.332039 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-workload-socket\") pod \"575d202b-f514-4312-8dc7-dfad51c6ca19\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " Nov 02 23:46:51.332291 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.332114 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-istio-data\") pod \"575d202b-f514-4312-8dc7-dfad51c6ca19\" (UID: \"575d202b-f514-4312-8dc7-dfad51c6ca19\") " Nov 02 23:46:51.332291 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.332255 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/575d202b-f514-4312-8dc7-dfad51c6ca19-istiod-ca-cert" pod "575d202b-f514-4312-8dc7-dfad51c6ca19" (UID: "575d202b-f514-4312-8dc7-dfad51c6ca19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:46:51.332638 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.332613 5727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/575d202b-f514-4312-8dc7-dfad51c6ca19-istiod-ca-cert\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:51.332915 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.332649 5727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-credential-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:51.332915 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.332617 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-istio-data" pod "575d202b-f514-4312-8dc7-dfad51c6ca19" (UID: "575d202b-f514-4312-8dc7-dfad51c6ca19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:46:51.332915 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.332669 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-workload-socket" pod "575d202b-f514-4312-8dc7-dfad51c6ca19" (UID: "575d202b-f514-4312-8dc7-dfad51c6ca19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:46:51.332915 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.332827 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-workload-certs" pod "575d202b-f514-4312-8dc7-dfad51c6ca19" (UID: "575d202b-f514-4312-8dc7-dfad51c6ca19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:46:51.334539 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.334512 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/575d202b-f514-4312-8dc7-dfad51c6ca19-kube-api-access-qddss" pod "575d202b-f514-4312-8dc7-dfad51c6ca19" (UID: "575d202b-f514-4312-8dc7-dfad51c6ca19"). InnerVolumeSpecName "kube-api-access-qddss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:46:51.335093 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.335067 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/575d202b-f514-4312-8dc7-dfad51c6ca19-istio-podinfo" pod "575d202b-f514-4312-8dc7-dfad51c6ca19" (UID: "575d202b-f514-4312-8dc7-dfad51c6ca19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:46:51.335214 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.335152 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-istio-envoy" pod "575d202b-f514-4312-8dc7-dfad51c6ca19" (UID: "575d202b-f514-4312-8dc7-dfad51c6ca19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:46:51.335321 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.335268 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/575d202b-f514-4312-8dc7-dfad51c6ca19-istio-token" pod "575d202b-f514-4312-8dc7-dfad51c6ca19" (UID: "575d202b-f514-4312-8dc7-dfad51c6ca19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:46:51.433593 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.433506 5727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-istio-envoy\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:51.433593 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.433544 5727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-workload-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:51.433593 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.433560 5727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/575d202b-f514-4312-8dc7-dfad51c6ca19-istio-podinfo\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:51.433593 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.433574 5727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-workload-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:51.433593 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.433588 5727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/575d202b-f514-4312-8dc7-dfad51c6ca19-istio-data\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:51.433879 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.433602 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qddss\" (UniqueName: \"kubernetes.io/projected/575d202b-f514-4312-8dc7-dfad51c6ca19-kube-api-access-qddss\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:51.433879 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.433618 5727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/575d202b-f514-4312-8dc7-dfad51c6ca19-istio-token\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:51.630564 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.630527 5727 generic.go:356] "Generic (PLEG): container finished" podID="575d202b-f514-4312-8dc7-dfad51c6ca19" containerID="983d66f724bc1f6dac51c13a78a6ac56816866652d92394fdee53a768d587f1b" exitCode=0 Nov 02 23:46:51.630740 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.630581 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" Nov 02 23:46:51.630740 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.630623 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" event={"ID":"575d202b-f514-4312-8dc7-dfad51c6ca19","Type":"ContainerDied","Data":"983d66f724bc1f6dac51c13a78a6ac56816866652d92394fdee53a768d587f1b"} Nov 02 23:46:51.630740 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.630724 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v" event={"ID":"575d202b-f514-4312-8dc7-dfad51c6ca19","Type":"ContainerDied","Data":"3bd096dd8f0cacaa1236efbe4eb2f6103a0d37bdb672033baf1671ea9cff0bc1"} Nov 02 23:46:51.630862 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.630752 5727 scope.go:122] "RemoveContainer" containerID="983d66f724bc1f6dac51c13a78a6ac56816866652d92394fdee53a768d587f1b" Nov 02 23:46:51.644857 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.644838 5727 scope.go:122] "RemoveContainer" containerID="983d66f724bc1f6dac51c13a78a6ac56816866652d92394fdee53a768d587f1b" Nov 02 23:46:51.645164 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:46:51.645119 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"983d66f724bc1f6dac51c13a78a6ac56816866652d92394fdee53a768d587f1b\": container with ID starting with 983d66f724bc1f6dac51c13a78a6ac56816866652d92394fdee53a768d587f1b not found: ID does not exist" containerID="983d66f724bc1f6dac51c13a78a6ac56816866652d92394fdee53a768d587f1b" Nov 02 23:46:51.645239 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.645174 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"983d66f724bc1f6dac51c13a78a6ac56816866652d92394fdee53a768d587f1b"} err="failed to get container status \"983d66f724bc1f6dac51c13a78a6ac56816866652d92394fdee53a768d587f1b\": rpc error: code = NotFound desc = could not find container \"983d66f724bc1f6dac51c13a78a6ac56816866652d92394fdee53a768d587f1b\": container with ID starting with 983d66f724bc1f6dac51c13a78a6ac56816866652d92394fdee53a768d587f1b not found: ID does not exist" Nov 02 23:46:51.796401 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.796316 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v"] Nov 02 23:46:51.799554 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.799527 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-c4m5v"] Nov 02 23:46:51.888389 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:51.888353 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="575d202b-f514-4312-8dc7-dfad51c6ca19" path="/var/lib/kubelet/pods/575d202b-f514-4312-8dc7-dfad51c6ca19/volumes" Nov 02 23:46:53.648210 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:53.648180 5727 generic.go:356] "Generic (PLEG): container finished" podID="e5415246-c0df-4f8d-a382-34a674b1f21f" containerID="477dc86b29f9b9ea4eea1c7c2b1ad30cd7391097ae1eb7d600fbfda318afd05e" exitCode=0 Nov 02 23:46:53.648658 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:53.648253 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" event={"ID":"e5415246-c0df-4f8d-a382-34a674b1f21f","Type":"ContainerDied","Data":"477dc86b29f9b9ea4eea1c7c2b1ad30cd7391097ae1eb7d600fbfda318afd05e"} Nov 02 23:46:53.801219 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:53.801161 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:46:53.859892 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:53.859858 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-workload-socket\") pod \"e5415246-c0df-4f8d-a382-34a674b1f21f\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " Nov 02 23:46:53.860022 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:53.859934 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-workload-certs\") pod \"e5415246-c0df-4f8d-a382-34a674b1f21f\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " Nov 02 23:46:53.860022 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:53.859973 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-istio-envoy\") pod \"e5415246-c0df-4f8d-a382-34a674b1f21f\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " Nov 02 23:46:53.860022 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:53.859999 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5415246-c0df-4f8d-a382-34a674b1f21f-istio-token\" (UniqueName: \"kubernetes.io/projected/e5415246-c0df-4f8d-a382-34a674b1f21f-istio-token\") pod \"e5415246-c0df-4f8d-a382-34a674b1f21f\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " Nov 02 23:46:53.860232 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:53.860038 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-credential-socket\") pod \"e5415246-c0df-4f8d-a382-34a674b1f21f\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " Nov 02 23:46:53.860232 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:53.860086 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5415246-c0df-4f8d-a382-34a674b1f21f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5415246-c0df-4f8d-a382-34a674b1f21f-istio-podinfo\") pod \"e5415246-c0df-4f8d-a382-34a674b1f21f\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " Nov 02 23:46:53.860232 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:53.860095 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-workload-socket" pod "e5415246-c0df-4f8d-a382-34a674b1f21f" (UID: "e5415246-c0df-4f8d-a382-34a674b1f21f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:46:53.860232 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:53.860157 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5415246-c0df-4f8d-a382-34a674b1f21f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5415246-c0df-4f8d-a382-34a674b1f21f-istiod-ca-cert\") pod \"e5415246-c0df-4f8d-a382-34a674b1f21f\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " Nov 02 23:46:53.860232 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:53.860198 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-workload-certs" pod "e5415246-c0df-4f8d-a382-34a674b1f21f" (UID: "e5415246-c0df-4f8d-a382-34a674b1f21f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:46:53.860536 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:53.860242 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-istio-data\") pod \"e5415246-c0df-4f8d-a382-34a674b1f21f\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " Nov 02 23:46:53.860536 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:53.860319 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5415246-c0df-4f8d-a382-34a674b1f21f-kube-api-access-zbhjg\" (UniqueName: \"kubernetes.io/projected/e5415246-c0df-4f8d-a382-34a674b1f21f-kube-api-access-zbhjg\") pod \"e5415246-c0df-4f8d-a382-34a674b1f21f\" (UID: \"e5415246-c0df-4f8d-a382-34a674b1f21f\") " Nov 02 23:46:53.860641 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:53.860562 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-istio-data" pod "e5415246-c0df-4f8d-a382-34a674b1f21f" (UID: "e5415246-c0df-4f8d-a382-34a674b1f21f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:46:53.860641 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:53.860622 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5415246-c0df-4f8d-a382-34a674b1f21f-istiod-ca-cert" pod "e5415246-c0df-4f8d-a382-34a674b1f21f" (UID: "e5415246-c0df-4f8d-a382-34a674b1f21f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:46:53.860754 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:53.860661 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-credential-socket" pod "e5415246-c0df-4f8d-a382-34a674b1f21f" (UID: "e5415246-c0df-4f8d-a382-34a674b1f21f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:46:53.861861 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:53.861057 5727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-istio-data\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:53.861861 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:53.861079 5727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-workload-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:53.861861 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:53.861093 5727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-workload-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:53.861861 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:53.861115 5727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-credential-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:53.861861 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:53.861173 5727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5415246-c0df-4f8d-a382-34a674b1f21f-istiod-ca-cert\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:53.862933 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:53.862903 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5415246-c0df-4f8d-a382-34a674b1f21f-istio-podinfo" pod "e5415246-c0df-4f8d-a382-34a674b1f21f" (UID: "e5415246-c0df-4f8d-a382-34a674b1f21f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:46:53.863019 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:53.862912 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-istio-envoy" pod "e5415246-c0df-4f8d-a382-34a674b1f21f" (UID: "e5415246-c0df-4f8d-a382-34a674b1f21f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:46:53.863111 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:53.863092 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5415246-c0df-4f8d-a382-34a674b1f21f-kube-api-access-zbhjg" pod "e5415246-c0df-4f8d-a382-34a674b1f21f" (UID: "e5415246-c0df-4f8d-a382-34a674b1f21f"). InnerVolumeSpecName "kube-api-access-zbhjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:46:53.863321 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:53.863299 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5415246-c0df-4f8d-a382-34a674b1f21f-istio-token" pod "e5415246-c0df-4f8d-a382-34a674b1f21f" (UID: "e5415246-c0df-4f8d-a382-34a674b1f21f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:46:53.962293 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:53.962263 5727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5415246-c0df-4f8d-a382-34a674b1f21f-istio-envoy\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:53.962293 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:53.962296 5727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5415246-c0df-4f8d-a382-34a674b1f21f-istio-token\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:53.962543 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:53.962313 5727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5415246-c0df-4f8d-a382-34a674b1f21f-istio-podinfo\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:53.962543 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:53.962330 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbhjg\" (UniqueName: \"kubernetes.io/projected/e5415246-c0df-4f8d-a382-34a674b1f21f-kube-api-access-zbhjg\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:54.657778 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:54.657749 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" Nov 02 23:46:54.658265 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:54.657783 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7" event={"ID":"e5415246-c0df-4f8d-a382-34a674b1f21f","Type":"ContainerDied","Data":"417b5ba77df971fbd7b85c5a633bd1bcea3e7c03b1662d8a1e0586fc45518a60"} Nov 02 23:46:54.658265 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:54.657860 5727 scope.go:122] "RemoveContainer" containerID="477dc86b29f9b9ea4eea1c7c2b1ad30cd7391097ae1eb7d600fbfda318afd05e" Nov 02 23:46:54.873094 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:54.873057 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7"] Nov 02 23:46:54.922957 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:54.922872 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-pmdg7"] Nov 02 23:46:55.574705 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:55.574677 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f49ab-557c48887f-mt5fg" Nov 02 23:46:55.886007 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:55.885920 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5415246-c0df-4f8d-a382-34a674b1f21f" path="/var/lib/kubelet/pods/e5415246-c0df-4f8d-a382-34a674b1f21f/volumes" Nov 02 23:46:57.846054 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:46:57.846021 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:46:58.127017 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:58.126942 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g"] Nov 02 23:46:58.129161 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:58.127639 5727 cpu_manager.go:414] "containerMap: removing container" podUID="2bad7bf4-87d2-4e0c-b18e-adb154740d49" containerName="cleanup" Nov 02 23:46:58.129161 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:58.127656 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bad7bf4-87d2-4e0c-b18e-adb154740d49" containerName="cleanup" Nov 02 23:46:58.129161 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:58.127667 5727 cpu_manager.go:414] "containerMap: removing container" podUID="e5415246-c0df-4f8d-a382-34a674b1f21f" containerName="istio-proxy" Nov 02 23:46:58.129161 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:58.127673 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5415246-c0df-4f8d-a382-34a674b1f21f" containerName="istio-proxy" Nov 02 23:46:58.129161 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:58.127684 5727 cpu_manager.go:414] "containerMap: removing container" podUID="575d202b-f514-4312-8dc7-dfad51c6ca19" containerName="istio-proxy" Nov 02 23:46:58.129161 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:58.127688 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="575d202b-f514-4312-8dc7-dfad51c6ca19" containerName="istio-proxy" Nov 02 23:46:58.129161 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:58.127792 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="575d202b-f514-4312-8dc7-dfad51c6ca19" containerName="istio-proxy" Nov 02 23:46:58.129161 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:58.127809 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5415246-c0df-4f8d-a382-34a674b1f21f" containerName="istio-proxy" Nov 02 23:46:58.131289 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:58.131269 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g" Nov 02 23:46:58.185069 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:58.185047 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-3058ee-serving-cert\"" Nov 02 23:46:58.211211 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:58.211178 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a75303-a945-49f6-9e10-1d9d5b25e6ea-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g\" (UID: \"20a75303-a945-49f6-9e10-1d9d5b25e6ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g" Nov 02 23:46:58.211318 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:58.211294 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpdbg\" (UniqueName: \"kubernetes.io/projected/20a75303-a945-49f6-9e10-1d9d5b25e6ea-kube-api-access-gpdbg\") pod \"maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g\" (UID: \"20a75303-a945-49f6-9e10-1d9d5b25e6ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g" Nov 02 23:46:58.211395 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:58.211374 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a75303-a945-49f6-9e10-1d9d5b25e6ea-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g\" (UID: \"20a75303-a945-49f6-9e10-1d9d5b25e6ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g" Nov 02 23:46:58.211443 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:58.211423 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20a75303-a945-49f6-9e10-1d9d5b25e6ea-metrics-certs\") pod \"maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g\" (UID: \"20a75303-a945-49f6-9e10-1d9d5b25e6ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g" Nov 02 23:46:58.237929 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:58.237905 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g"] Nov 02 23:46:58.312925 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:58.312896 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a75303-a945-49f6-9e10-1d9d5b25e6ea-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g\" (UID: \"20a75303-a945-49f6-9e10-1d9d5b25e6ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g" Nov 02 23:46:58.313102 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:58.312983 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpdbg\" (UniqueName: \"kubernetes.io/projected/20a75303-a945-49f6-9e10-1d9d5b25e6ea-kube-api-access-gpdbg\") pod \"maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g\" (UID: \"20a75303-a945-49f6-9e10-1d9d5b25e6ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g" Nov 02 23:46:58.313102 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:58.313014 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a75303-a945-49f6-9e10-1d9d5b25e6ea-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g\" (UID: \"20a75303-a945-49f6-9e10-1d9d5b25e6ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g" Nov 02 23:46:58.313102 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:58.313048 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20a75303-a945-49f6-9e10-1d9d5b25e6ea-metrics-certs\") pod \"maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g\" (UID: \"20a75303-a945-49f6-9e10-1d9d5b25e6ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g" Nov 02 23:46:58.313102 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:46:58.313054 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-3058ee-serving-cert: secret "maas-api-e2e-extmod-a-3058ee-serving-cert" not found Nov 02 23:46:58.313377 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:46:58.313151 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20a75303-a945-49f6-9e10-1d9d5b25e6ea-maas-api-tls podName:20a75303-a945-49f6-9e10-1d9d5b25e6ea nodeName:}" failed. No retries permitted until 2026-11-02 23:46:58.81311074 +0000 UTC m=+3505.051608381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/20a75303-a945-49f6-9e10-1d9d5b25e6ea-maas-api-tls") pod "maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g" (UID: "20a75303-a945-49f6-9e10-1d9d5b25e6ea") : secret "maas-api-e2e-extmod-a-3058ee-serving-cert" not found Nov 02 23:46:58.313766 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:58.313740 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a75303-a945-49f6-9e10-1d9d5b25e6ea-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g\" (UID: \"20a75303-a945-49f6-9e10-1d9d5b25e6ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g" Nov 02 23:46:58.315858 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:58.315838 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20a75303-a945-49f6-9e10-1d9d5b25e6ea-metrics-certs\") pod \"maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g\" (UID: \"20a75303-a945-49f6-9e10-1d9d5b25e6ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g" Nov 02 23:46:58.378952 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:58.378875 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpdbg\" (UniqueName: \"kubernetes.io/projected/20a75303-a945-49f6-9e10-1d9d5b25e6ea-kube-api-access-gpdbg\") pod \"maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g\" (UID: \"20a75303-a945-49f6-9e10-1d9d5b25e6ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g" Nov 02 23:46:58.819774 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:58.819739 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a75303-a945-49f6-9e10-1d9d5b25e6ea-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g\" (UID: \"20a75303-a945-49f6-9e10-1d9d5b25e6ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g" Nov 02 23:46:58.822419 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:58.822395 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a75303-a945-49f6-9e10-1d9d5b25e6ea-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g\" (UID: \"20a75303-a945-49f6-9e10-1d9d5b25e6ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g" Nov 02 23:46:59.045355 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:59.045318 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g" Nov 02 23:46:59.431713 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:59.431568 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g"] Nov 02 23:46:59.435169 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:46:59.435101 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a75303_a945_49f6_9e10_1d9d5b25e6ea.slice/crio-8dc39a9dfdb32771bbf2ed44e9cbaa24ecfa4769b4de53f5a3404b036605b15f WatchSource:0}: Error finding container 8dc39a9dfdb32771bbf2ed44e9cbaa24ecfa4769b4de53f5a3404b036605b15f: Status 404 returned error can't find the container with id 8dc39a9dfdb32771bbf2ed44e9cbaa24ecfa4769b4de53f5a3404b036605b15f Nov 02 23:46:59.437055 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:59.437035 5727 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 23:46:59.698823 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:46:59.698722 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g" event={"ID":"20a75303-a945-49f6-9e10-1d9d5b25e6ea","Type":"ContainerStarted","Data":"8dc39a9dfdb32771bbf2ed44e9cbaa24ecfa4769b4de53f5a3404b036605b15f"} Nov 02 23:47:00.708007 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:47:00.707908 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g" event={"ID":"20a75303-a945-49f6-9e10-1d9d5b25e6ea","Type":"ContainerStarted","Data":"08a217c6d2732d3706839d5a89530cc4791d82957e26ab5b6ed67dc691f01a81"} Nov 02 23:47:00.708521 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:47:00.708062 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g" Nov 02 23:47:00.855945 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:47:00.855894 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g" podStartSLOduration=1.945789246 podStartE2EDuration="2.855880457s" totalImagesPullingTime="910.091211ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:47:00.853350751 +0000 UTC m=+3507.091848410" watchObservedRunningTime="2026-11-02 23:47:00.855880457 +0000 UTC m=+3507.094378130" Nov 02 23:47:06.722829 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:47:06.722798 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g" Nov 02 23:47:23.506734 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:47:23.506703 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg"] Nov 02 23:47:23.511070 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:47:23.511053 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg" Nov 02 23:47:23.565510 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:47:23.565485 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-1498df-serving-cert\"" Nov 02 23:47:23.610637 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:47:23.610597 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1986d8d6-894f-49e7-a357-24fff971f75b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg\" (UID: \"1986d8d6-894f-49e7-a357-24fff971f75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg" Nov 02 23:47:23.610805 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:47:23.610682 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1986d8d6-894f-49e7-a357-24fff971f75b-metrics-certs\") pod \"maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg\" (UID: \"1986d8d6-894f-49e7-a357-24fff971f75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg" Nov 02 23:47:23.610805 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:47:23.610773 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1986d8d6-894f-49e7-a357-24fff971f75b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg\" (UID: \"1986d8d6-894f-49e7-a357-24fff971f75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg" Nov 02 23:47:23.610965 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:47:23.610825 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzjt4\" (UniqueName: \"kubernetes.io/projected/1986d8d6-894f-49e7-a357-24fff971f75b-kube-api-access-fzjt4\") pod \"maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg\" (UID: \"1986d8d6-894f-49e7-a357-24fff971f75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg" Nov 02 23:47:23.617882 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:47:23.617847 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg"] Nov 02 23:47:23.712218 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:47:23.712188 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1986d8d6-894f-49e7-a357-24fff971f75b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg\" (UID: \"1986d8d6-894f-49e7-a357-24fff971f75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg" Nov 02 23:47:23.712401 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:47:23.712267 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzjt4\" (UniqueName: \"kubernetes.io/projected/1986d8d6-894f-49e7-a357-24fff971f75b-kube-api-access-fzjt4\") pod \"maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg\" (UID: \"1986d8d6-894f-49e7-a357-24fff971f75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg" Nov 02 23:47:23.712401 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:47:23.712302 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1986d8d6-894f-49e7-a357-24fff971f75b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg\" (UID: \"1986d8d6-894f-49e7-a357-24fff971f75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg" Nov 02 23:47:23.712401 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:47:23.712331 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-1498df-serving-cert: secret "maas-api-e2e-extmod-a-1498df-serving-cert" not found Nov 02 23:47:23.712401 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:47:23.712350 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1986d8d6-894f-49e7-a357-24fff971f75b-metrics-certs\") pod \"maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg\" (UID: \"1986d8d6-894f-49e7-a357-24fff971f75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg" Nov 02 23:47:23.712401 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:47:23.712389 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1986d8d6-894f-49e7-a357-24fff971f75b-maas-api-tls podName:1986d8d6-894f-49e7-a357-24fff971f75b nodeName:}" failed. No retries permitted until 2026-11-02 23:47:24.212374167 +0000 UTC m=+3530.450871821 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1986d8d6-894f-49e7-a357-24fff971f75b-maas-api-tls") pod "maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg" (UID: "1986d8d6-894f-49e7-a357-24fff971f75b") : secret "maas-api-e2e-extmod-a-1498df-serving-cert" not found Nov 02 23:47:23.713052 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:47:23.713034 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1986d8d6-894f-49e7-a357-24fff971f75b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg\" (UID: \"1986d8d6-894f-49e7-a357-24fff971f75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg" Nov 02 23:47:23.714962 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:47:23.714936 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1986d8d6-894f-49e7-a357-24fff971f75b-metrics-certs\") pod \"maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg\" (UID: \"1986d8d6-894f-49e7-a357-24fff971f75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg" Nov 02 23:47:23.775700 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:47:23.775628 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzjt4\" (UniqueName: \"kubernetes.io/projected/1986d8d6-894f-49e7-a357-24fff971f75b-kube-api-access-fzjt4\") pod \"maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg\" (UID: \"1986d8d6-894f-49e7-a357-24fff971f75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg" Nov 02 23:47:24.218522 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:47:24.218488 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1986d8d6-894f-49e7-a357-24fff971f75b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg\" (UID: \"1986d8d6-894f-49e7-a357-24fff971f75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg" Nov 02 23:47:24.221233 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:47:24.221199 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1986d8d6-894f-49e7-a357-24fff971f75b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg\" (UID: \"1986d8d6-894f-49e7-a357-24fff971f75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg" Nov 02 23:47:24.425304 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:47:24.425276 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg" Nov 02 23:47:24.810262 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:47:24.810228 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg"] Nov 02 23:47:24.810943 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:47:24.810919 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1986d8d6_894f_49e7_a357_24fff971f75b.slice/crio-80a75cd3abe329118889b234b535beb1260848c3bd8217a53115ef00aa4c822f WatchSource:0}: Error finding container 80a75cd3abe329118889b234b535beb1260848c3bd8217a53115ef00aa4c822f: Status 404 returned error can't find the container with id 80a75cd3abe329118889b234b535beb1260848c3bd8217a53115ef00aa4c822f Nov 02 23:47:24.897688 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:47:24.897657 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg" event={"ID":"1986d8d6-894f-49e7-a357-24fff971f75b","Type":"ContainerStarted","Data":"80a75cd3abe329118889b234b535beb1260848c3bd8217a53115ef00aa4c822f"} Nov 02 23:47:25.905537 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:47:25.905444 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg" event={"ID":"1986d8d6-894f-49e7-a357-24fff971f75b","Type":"ContainerStarted","Data":"1c89cc0fc40ff9c1fdf63ce9a7f925c568528d2ac7d622ebe456e5fe3c67d2f6"} Nov 02 23:47:25.905537 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:47:25.905501 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg" Nov 02 23:47:26.023403 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:47:26.023356 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg" podStartSLOduration=2.189341389 podStartE2EDuration="3.02334152s" totalImagesPullingTime="834.000131ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:47:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:47:26.021754787 +0000 UTC m=+3532.260252474" watchObservedRunningTime="2026-11-02 23:47:26.02334152 +0000 UTC m=+3532.261839181" Nov 02 23:47:27.847334 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:47:27.847290 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:47:31.921752 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:47:31.921720 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg" Nov 02 23:47:57.846522 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:47:57.846485 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:48:27.845932 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:48:27.845897 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:48:34.733216 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:34.733183 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 02 23:48:34.747991 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:34.747969 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 23:48:34.807537 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:34.807508 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 02 23:48:34.826745 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:34.826714 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 23:48:40.039523 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:40.039492 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-942137-5c7c58df55-7r9vb"] Nov 02 23:48:40.043952 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:40.043931 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-942137-5c7c58df55-7r9vb" Nov 02 23:48:40.097931 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:40.097899 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-942137\"" Nov 02 23:48:40.102762 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:40.102732 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-942137-5c7c58df55-7r9vb"] Nov 02 23:48:40.109094 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:40.108732 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts5k2\" (UniqueName: \"kubernetes.io/projected/a5ad8962-82c6-4113-933e-9d295480223d-kube-api-access-ts5k2\") pod \"payload-pre-processing-e2e-extmod-b-942137-5c7c58df55-7r9vb\" (UID: \"a5ad8962-82c6-4113-933e-9d295480223d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-942137-5c7c58df55-7r9vb" Nov 02 23:48:40.109094 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:40.108892 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5ad8962-82c6-4113-933e-9d295480223d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-942137-5c7c58df55-7r9vb\" (UID: \"a5ad8962-82c6-4113-933e-9d295480223d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-942137-5c7c58df55-7r9vb" Nov 02 23:48:40.149453 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:40.149413 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-942137-dockercfg-c9vkh\"" Nov 02 23:48:40.189499 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:40.189442 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-942137-6d5cc6b689-wjbdr"] Nov 02 23:48:40.194977 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:40.194953 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-942137-6d5cc6b689-wjbdr" Nov 02 23:48:40.210216 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:40.210181 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts5k2\" (UniqueName: \"kubernetes.io/projected/a5ad8962-82c6-4113-933e-9d295480223d-kube-api-access-ts5k2\") pod \"payload-pre-processing-e2e-extmod-b-942137-5c7c58df55-7r9vb\" (UID: \"a5ad8962-82c6-4113-933e-9d295480223d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-942137-5c7c58df55-7r9vb" Nov 02 23:48:40.210547 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:40.210522 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5ad8962-82c6-4113-933e-9d295480223d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-942137-5c7c58df55-7r9vb\" (UID: \"a5ad8962-82c6-4113-933e-9d295480223d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-942137-5c7c58df55-7r9vb" Nov 02 23:48:40.211180 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:40.211122 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5ad8962-82c6-4113-933e-9d295480223d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-942137-5c7c58df55-7r9vb\" (UID: \"a5ad8962-82c6-4113-933e-9d295480223d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-942137-5c7c58df55-7r9vb" Nov 02 23:48:40.249018 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:40.248990 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-942137-6d5cc6b689-wjbdr"] Nov 02 23:48:40.271255 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:40.271220 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts5k2\" (UniqueName: \"kubernetes.io/projected/a5ad8962-82c6-4113-933e-9d295480223d-kube-api-access-ts5k2\") pod \"payload-pre-processing-e2e-extmod-b-942137-5c7c58df55-7r9vb\" (UID: \"a5ad8962-82c6-4113-933e-9d295480223d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-942137-5c7c58df55-7r9vb" Nov 02 23:48:40.311763 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:40.311676 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb5397a3-06aa-4c5e-ac18-9b3411a753d0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-942137-6d5cc6b689-wjbdr\" (UID: \"eb5397a3-06aa-4c5e-ac18-9b3411a753d0\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-942137-6d5cc6b689-wjbdr" Nov 02 23:48:40.311926 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:40.311819 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b459\" (UniqueName: \"kubernetes.io/projected/eb5397a3-06aa-4c5e-ac18-9b3411a753d0-kube-api-access-6b459\") pod \"payload-processing-e2e-extmod-b-942137-6d5cc6b689-wjbdr\" (UID: \"eb5397a3-06aa-4c5e-ac18-9b3411a753d0\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-942137-6d5cc6b689-wjbdr" Nov 02 23:48:40.360372 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:40.360322 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-942137-5c7c58df55-7r9vb" Nov 02 23:48:40.413830 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:40.413790 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb5397a3-06aa-4c5e-ac18-9b3411a753d0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-942137-6d5cc6b689-wjbdr\" (UID: \"eb5397a3-06aa-4c5e-ac18-9b3411a753d0\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-942137-6d5cc6b689-wjbdr" Nov 02 23:48:40.414034 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:40.413918 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b459\" (UniqueName: \"kubernetes.io/projected/eb5397a3-06aa-4c5e-ac18-9b3411a753d0-kube-api-access-6b459\") pod \"payload-processing-e2e-extmod-b-942137-6d5cc6b689-wjbdr\" (UID: \"eb5397a3-06aa-4c5e-ac18-9b3411a753d0\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-942137-6d5cc6b689-wjbdr" Nov 02 23:48:40.415014 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:40.414984 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb5397a3-06aa-4c5e-ac18-9b3411a753d0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-942137-6d5cc6b689-wjbdr\" (UID: \"eb5397a3-06aa-4c5e-ac18-9b3411a753d0\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-942137-6d5cc6b689-wjbdr" Nov 02 23:48:40.475831 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:40.475794 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b459\" (UniqueName: \"kubernetes.io/projected/eb5397a3-06aa-4c5e-ac18-9b3411a753d0-kube-api-access-6b459\") pod \"payload-processing-e2e-extmod-b-942137-6d5cc6b689-wjbdr\" (UID: \"eb5397a3-06aa-4c5e-ac18-9b3411a753d0\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-942137-6d5cc6b689-wjbdr" Nov 02 23:48:40.510786 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:40.510746 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-942137-6d5cc6b689-wjbdr" Nov 02 23:48:40.767645 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:40.767613 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-942137-5c7c58df55-7r9vb"] Nov 02 23:48:40.769057 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:48:40.769031 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5ad8962_82c6_4113_933e_9d295480223d.slice/crio-343c6bdc7ff42f2f91854c0f85c848ba8bc6216083219dfd18487cb67702fbf7 WatchSource:0}: Error finding container 343c6bdc7ff42f2f91854c0f85c848ba8bc6216083219dfd18487cb67702fbf7: Status 404 returned error can't find the container with id 343c6bdc7ff42f2f91854c0f85c848ba8bc6216083219dfd18487cb67702fbf7 Nov 02 23:48:40.911378 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:40.911347 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-942137-6d5cc6b689-wjbdr"] Nov 02 23:48:40.911742 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:48:40.911717 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb5397a3_06aa_4c5e_ac18_9b3411a753d0.slice/crio-70d16fca209bc8e5224d9a6bd5404f2e02f2ace6d5b2f6f0fd16b07b0fa87355 WatchSource:0}: Error finding container 70d16fca209bc8e5224d9a6bd5404f2e02f2ace6d5b2f6f0fd16b07b0fa87355: Status 404 returned error can't find the container with id 70d16fca209bc8e5224d9a6bd5404f2e02f2ace6d5b2f6f0fd16b07b0fa87355 Nov 02 23:48:41.552346 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:41.552298 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-942137-5c7c58df55-7r9vb" event={"ID":"a5ad8962-82c6-4113-933e-9d295480223d","Type":"ContainerStarted","Data":"cb2e457a6622134b08d9b5ccd67026da9ea9e57113f4770187105b76e4fd7662"} Nov 02 23:48:41.552793 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:41.552700 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-942137-5c7c58df55-7r9vb" event={"ID":"a5ad8962-82c6-4113-933e-9d295480223d","Type":"ContainerStarted","Data":"343c6bdc7ff42f2f91854c0f85c848ba8bc6216083219dfd18487cb67702fbf7"} Nov 02 23:48:41.552793 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:41.552742 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-942137-5c7c58df55-7r9vb" Nov 02 23:48:41.554642 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:41.554614 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-942137-6d5cc6b689-wjbdr" event={"ID":"eb5397a3-06aa-4c5e-ac18-9b3411a753d0","Type":"ContainerStarted","Data":"8c8f45c74d3711a5d7531cdb0a19708ff11076d33dd139021e2e7661e506d591"} Nov 02 23:48:41.554812 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:41.554659 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-942137-6d5cc6b689-wjbdr" event={"ID":"eb5397a3-06aa-4c5e-ac18-9b3411a753d0","Type":"ContainerStarted","Data":"70d16fca209bc8e5224d9a6bd5404f2e02f2ace6d5b2f6f0fd16b07b0fa87355"} Nov 02 23:48:41.554812 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:41.554690 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-942137-6d5cc6b689-wjbdr" Nov 02 23:48:41.672172 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:41.670648 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-942137-5c7c58df55-7r9vb" podStartSLOduration=2.670629795 podStartE2EDuration="2.670629795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:48:41.666742688 +0000 UTC m=+3607.905240375" watchObservedRunningTime="2026-11-02 23:48:41.670629795 +0000 UTC m=+3607.909127458" Nov 02 23:48:41.786911 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:41.786856 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-942137-6d5cc6b689-wjbdr" podStartSLOduration=1.786838119 podStartE2EDuration="1.786838119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:48:41.782217934 +0000 UTC m=+3608.020715654" watchObservedRunningTime="2026-11-02 23:48:41.786838119 +0000 UTC m=+3608.025335783" Nov 02 23:48:52.567961 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:52.567876 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-942137-6d5cc6b689-wjbdr" Nov 02 23:48:52.568477 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:48:52.567964 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-942137-5c7c58df55-7r9vb" Nov 02 23:48:57.847923 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:48:57.847881 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:49:27.845972 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:49:27.845932 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:49:44.946467 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:49:44.946434 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0f49ab-557c48887f-mt5fg"] Nov 02 23:49:49.947369 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:49:49.947330 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f49ab-557c48887f-mt5fg" podUID="453c1e89-68c1-4883-ba1b-d2a19c6b2dfb" containerName="payload-processing" containerID="cri-o://06f07e95b4569b4252c03167cf012a9e817c4d5658a9d3c1567152fb3ea8bc46" gracePeriod=25 Nov 02 23:49:50.102075 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:49:50.102042 5727 generic.go:356] "Generic (PLEG): container finished" podID="453c1e89-68c1-4883-ba1b-d2a19c6b2dfb" containerID="06f07e95b4569b4252c03167cf012a9e817c4d5658a9d3c1567152fb3ea8bc46" exitCode=0 Nov 02 23:49:50.102285 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:49:50.102149 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f49ab-557c48887f-mt5fg" event={"ID":"453c1e89-68c1-4883-ba1b-d2a19c6b2dfb","Type":"ContainerDied","Data":"06f07e95b4569b4252c03167cf012a9e817c4d5658a9d3c1567152fb3ea8bc46"} Nov 02 23:49:50.218963 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:49:50.218933 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f49ab-557c48887f-mt5fg" Nov 02 23:49:50.329848 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:49:50.329818 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/453c1e89-68c1-4883-ba1b-d2a19c6b2dfb-kube-api-access-dps79\" (UniqueName: \"kubernetes.io/projected/453c1e89-68c1-4883-ba1b-d2a19c6b2dfb-kube-api-access-dps79\") pod \"453c1e89-68c1-4883-ba1b-d2a19c6b2dfb\" (UID: \"453c1e89-68c1-4883-ba1b-d2a19c6b2dfb\") " Nov 02 23:49:50.330181 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:49:50.329935 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/453c1e89-68c1-4883-ba1b-d2a19c6b2dfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/453c1e89-68c1-4883-ba1b-d2a19c6b2dfb-plugins-config-volume\") pod \"453c1e89-68c1-4883-ba1b-d2a19c6b2dfb\" (UID: \"453c1e89-68c1-4883-ba1b-d2a19c6b2dfb\") " Nov 02 23:49:50.330297 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:49:50.330273 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453c1e89-68c1-4883-ba1b-d2a19c6b2dfb-plugins-config-volume" pod "453c1e89-68c1-4883-ba1b-d2a19c6b2dfb" (UID: "453c1e89-68c1-4883-ba1b-d2a19c6b2dfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:49:50.332245 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:49:50.332224 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453c1e89-68c1-4883-ba1b-d2a19c6b2dfb-kube-api-access-dps79" pod "453c1e89-68c1-4883-ba1b-d2a19c6b2dfb" (UID: "453c1e89-68c1-4883-ba1b-d2a19c6b2dfb"). InnerVolumeSpecName "kube-api-access-dps79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:49:50.430968 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:49:50.430929 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dps79\" (UniqueName: \"kubernetes.io/projected/453c1e89-68c1-4883-ba1b-d2a19c6b2dfb-kube-api-access-dps79\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:49:50.430968 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:49:50.430961 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/453c1e89-68c1-4883-ba1b-d2a19c6b2dfb-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:49:51.110304 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:49:51.110263 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f49ab-557c48887f-mt5fg" event={"ID":"453c1e89-68c1-4883-ba1b-d2a19c6b2dfb","Type":"ContainerDied","Data":"2edc951fed7e6c5f8b1b09b9f95c234d3ba6a57164f16a151574b7ccb074f487"} Nov 02 23:49:51.110780 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:49:51.110330 5727 scope.go:122] "RemoveContainer" containerID="06f07e95b4569b4252c03167cf012a9e817c4d5658a9d3c1567152fb3ea8bc46" Nov 02 23:49:51.110780 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:49:51.110271 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f49ab-557c48887f-mt5fg" Nov 02 23:49:51.274637 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:49:51.274603 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0f49ab-557c48887f-mt5fg"] Nov 02 23:49:51.323895 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:49:51.323864 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0f49ab-557c48887f-mt5fg"] Nov 02 23:49:51.883533 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:49:51.883501 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="453c1e89-68c1-4883-ba1b-d2a19c6b2dfb" path="/var/lib/kubelet/pods/453c1e89-68c1-4883-ba1b-d2a19c6b2dfb/volumes" Nov 02 23:49:57.846357 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:49:57.846322 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:50:03.206370 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:03.206337 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-942137-7b896967c-j7p9z"] Nov 02 23:50:03.206954 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:03.206941 5727 cpu_manager.go:414] "containerMap: removing container" podUID="453c1e89-68c1-4883-ba1b-d2a19c6b2dfb" containerName="payload-processing" Nov 02 23:50:03.207012 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:03.206955 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="453c1e89-68c1-4883-ba1b-d2a19c6b2dfb" containerName="payload-processing" Nov 02 23:50:03.207098 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:03.207089 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="453c1e89-68c1-4883-ba1b-d2a19c6b2dfb" containerName="payload-processing" Nov 02 23:50:03.212038 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:03.212018 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-942137-7b896967c-j7p9z" Nov 02 23:50:03.267305 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:03.267276 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-942137-7b896967c-j7p9z"] Nov 02 23:50:03.267448 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:03.267412 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/163e0b93-962f-4506-9465-c6ed52fe99b4-metrics-certs\") pod \"maas-api-e2e-extmod-b-942137-7b896967c-j7p9z\" (UID: \"163e0b93-962f-4506-9465-c6ed52fe99b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-942137-7b896967c-j7p9z" Nov 02 23:50:03.267566 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:03.267551 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/163e0b93-962f-4506-9465-c6ed52fe99b4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-942137-7b896967c-j7p9z\" (UID: \"163e0b93-962f-4506-9465-c6ed52fe99b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-942137-7b896967c-j7p9z" Nov 02 23:50:03.267657 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:03.267642 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/163e0b93-962f-4506-9465-c6ed52fe99b4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-942137-7b896967c-j7p9z\" (UID: \"163e0b93-962f-4506-9465-c6ed52fe99b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-942137-7b896967c-j7p9z" Nov 02 23:50:03.267737 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:03.267718 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkkc4\" (UniqueName: \"kubernetes.io/projected/163e0b93-962f-4506-9465-c6ed52fe99b4-kube-api-access-qkkc4\") pod \"maas-api-e2e-extmod-b-942137-7b896967c-j7p9z\" (UID: \"163e0b93-962f-4506-9465-c6ed52fe99b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-942137-7b896967c-j7p9z" Nov 02 23:50:03.312770 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:03.312739 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-942137-serving-cert\"" Nov 02 23:50:03.369100 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:03.369066 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/163e0b93-962f-4506-9465-c6ed52fe99b4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-942137-7b896967c-j7p9z\" (UID: \"163e0b93-962f-4506-9465-c6ed52fe99b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-942137-7b896967c-j7p9z" Nov 02 23:50:03.369283 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:03.369177 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/163e0b93-962f-4506-9465-c6ed52fe99b4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-942137-7b896967c-j7p9z\" (UID: \"163e0b93-962f-4506-9465-c6ed52fe99b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-942137-7b896967c-j7p9z" Nov 02 23:50:03.369283 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:03.369196 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkkc4\" (UniqueName: \"kubernetes.io/projected/163e0b93-962f-4506-9465-c6ed52fe99b4-kube-api-access-qkkc4\") pod \"maas-api-e2e-extmod-b-942137-7b896967c-j7p9z\" (UID: \"163e0b93-962f-4506-9465-c6ed52fe99b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-942137-7b896967c-j7p9z" Nov 02 23:50:03.369283 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:50:03.369249 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-942137-serving-cert: secret "maas-api-e2e-extmod-b-942137-serving-cert" not found Nov 02 23:50:03.369283 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:03.369267 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/163e0b93-962f-4506-9465-c6ed52fe99b4-metrics-certs\") pod \"maas-api-e2e-extmod-b-942137-7b896967c-j7p9z\" (UID: \"163e0b93-962f-4506-9465-c6ed52fe99b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-942137-7b896967c-j7p9z" Nov 02 23:50:03.369519 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:50:03.369322 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/163e0b93-962f-4506-9465-c6ed52fe99b4-maas-api-tls podName:163e0b93-962f-4506-9465-c6ed52fe99b4 nodeName:}" failed. No retries permitted until 2026-11-02 23:50:03.869301718 +0000 UTC m=+3690.107799376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/163e0b93-962f-4506-9465-c6ed52fe99b4-maas-api-tls") pod "maas-api-e2e-extmod-b-942137-7b896967c-j7p9z" (UID: "163e0b93-962f-4506-9465-c6ed52fe99b4") : secret "maas-api-e2e-extmod-b-942137-serving-cert" not found Nov 02 23:50:03.369876 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:03.369855 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/163e0b93-962f-4506-9465-c6ed52fe99b4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-942137-7b896967c-j7p9z\" (UID: \"163e0b93-962f-4506-9465-c6ed52fe99b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-942137-7b896967c-j7p9z" Nov 02 23:50:03.371856 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:03.371831 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/163e0b93-962f-4506-9465-c6ed52fe99b4-metrics-certs\") pod \"maas-api-e2e-extmod-b-942137-7b896967c-j7p9z\" (UID: \"163e0b93-962f-4506-9465-c6ed52fe99b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-942137-7b896967c-j7p9z" Nov 02 23:50:03.432061 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:03.432037 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkkc4\" (UniqueName: \"kubernetes.io/projected/163e0b93-962f-4506-9465-c6ed52fe99b4-kube-api-access-qkkc4\") pod \"maas-api-e2e-extmod-b-942137-7b896967c-j7p9z\" (UID: \"163e0b93-962f-4506-9465-c6ed52fe99b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-942137-7b896967c-j7p9z" Nov 02 23:50:03.875502 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:03.875429 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/163e0b93-962f-4506-9465-c6ed52fe99b4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-942137-7b896967c-j7p9z\" (UID: \"163e0b93-962f-4506-9465-c6ed52fe99b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-942137-7b896967c-j7p9z" Nov 02 23:50:03.878806 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:03.878777 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/163e0b93-962f-4506-9465-c6ed52fe99b4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-942137-7b896967c-j7p9z\" (UID: \"163e0b93-962f-4506-9465-c6ed52fe99b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-942137-7b896967c-j7p9z" Nov 02 23:50:04.127471 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:04.127384 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-942137-7b896967c-j7p9z" Nov 02 23:50:04.509572 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:04.509549 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-942137-7b896967c-j7p9z"] Nov 02 23:50:04.511233 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:50:04.511207 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163e0b93_962f_4506_9465_c6ed52fe99b4.slice/crio-b15741ccb288f4c654b7b5c0ff16d03a73394875fa22eb7e487982b4287c043b WatchSource:0}: Error finding container b15741ccb288f4c654b7b5c0ff16d03a73394875fa22eb7e487982b4287c043b: Status 404 returned error can't find the container with id b15741ccb288f4c654b7b5c0ff16d03a73394875fa22eb7e487982b4287c043b Nov 02 23:50:05.232312 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:05.232280 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-942137-7b896967c-j7p9z" event={"ID":"163e0b93-962f-4506-9465-c6ed52fe99b4","Type":"ContainerStarted","Data":"b15741ccb288f4c654b7b5c0ff16d03a73394875fa22eb7e487982b4287c043b"} Nov 02 23:50:06.241795 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:06.241759 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-942137-7b896967c-j7p9z" event={"ID":"163e0b93-962f-4506-9465-c6ed52fe99b4","Type":"ContainerStarted","Data":"a7a89e6d9a0606e09fa0c64f58d742bfdcfc5248d20041dc450068513242be75"} Nov 02 23:50:06.242223 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:06.241847 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-942137-7b896967c-j7p9z" Nov 02 23:50:06.358418 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:06.358365 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-942137-7b896967c-j7p9z" podStartSLOduration=2.475161222 podStartE2EDuration="3.358351741s" totalImagesPullingTime="883.190519ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:50:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:50:06.357824437 +0000 UTC m=+3692.596322117" watchObservedRunningTime="2026-11-02 23:50:06.358351741 +0000 UTC m=+3692.596849442" Nov 02 23:50:12.254953 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:12.254920 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-942137-7b896967c-j7p9z" Nov 02 23:50:13.133800 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:13.133765 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6"] Nov 02 23:50:13.138219 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:13.138204 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6" Nov 02 23:50:13.240510 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:13.240478 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-d75a69-serving-cert\"" Nov 02 23:50:13.244199 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:13.244178 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6"] Nov 02 23:50:13.285106 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:13.285035 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e561e563-f494-44d6-bced-7db92ae72bb9-metrics-certs\") pod \"maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6\" (UID: \"e561e563-f494-44d6-bced-7db92ae72bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6" Nov 02 23:50:13.285495 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:13.285182 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e561e563-f494-44d6-bced-7db92ae72bb9-maas-api-tls\") pod \"maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6\" (UID: \"e561e563-f494-44d6-bced-7db92ae72bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6" Nov 02 23:50:13.285495 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:13.285219 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e561e563-f494-44d6-bced-7db92ae72bb9-openshift-service-ca\") pod \"maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6\" (UID: \"e561e563-f494-44d6-bced-7db92ae72bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6" Nov 02 23:50:13.285495 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:13.285241 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxt9t\" (UniqueName: \"kubernetes.io/projected/e561e563-f494-44d6-bced-7db92ae72bb9-kube-api-access-wxt9t\") pod \"maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6\" (UID: \"e561e563-f494-44d6-bced-7db92ae72bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6" Nov 02 23:50:13.386230 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:13.386149 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e561e563-f494-44d6-bced-7db92ae72bb9-metrics-certs\") pod \"maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6\" (UID: \"e561e563-f494-44d6-bced-7db92ae72bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6" Nov 02 23:50:13.386230 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:13.386221 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e561e563-f494-44d6-bced-7db92ae72bb9-maas-api-tls\") pod \"maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6\" (UID: \"e561e563-f494-44d6-bced-7db92ae72bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6" Nov 02 23:50:13.386469 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:13.386238 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e561e563-f494-44d6-bced-7db92ae72bb9-openshift-service-ca\") pod \"maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6\" (UID: \"e561e563-f494-44d6-bced-7db92ae72bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6" Nov 02 23:50:13.386469 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:50:13.386328 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-d75a69-serving-cert: secret "maas-api-e2e-explicit-d75a69-serving-cert" not found Nov 02 23:50:13.386469 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:13.386352 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxt9t\" (UniqueName: \"kubernetes.io/projected/e561e563-f494-44d6-bced-7db92ae72bb9-kube-api-access-wxt9t\") pod \"maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6\" (UID: \"e561e563-f494-44d6-bced-7db92ae72bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6" Nov 02 23:50:13.386469 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:50:13.386409 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e561e563-f494-44d6-bced-7db92ae72bb9-maas-api-tls podName:e561e563-f494-44d6-bced-7db92ae72bb9 nodeName:}" failed. No retries permitted until 2026-11-02 23:50:13.886388711 +0000 UTC m=+3700.124886366 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e561e563-f494-44d6-bced-7db92ae72bb9-maas-api-tls") pod "maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6" (UID: "e561e563-f494-44d6-bced-7db92ae72bb9") : secret "maas-api-e2e-explicit-d75a69-serving-cert" not found Nov 02 23:50:13.386896 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:13.386878 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e561e563-f494-44d6-bced-7db92ae72bb9-openshift-service-ca\") pod \"maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6\" (UID: \"e561e563-f494-44d6-bced-7db92ae72bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6" Nov 02 23:50:13.388819 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:13.388794 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e561e563-f494-44d6-bced-7db92ae72bb9-metrics-certs\") pod \"maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6\" (UID: \"e561e563-f494-44d6-bced-7db92ae72bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6" Nov 02 23:50:13.449554 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:13.449521 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxt9t\" (UniqueName: \"kubernetes.io/projected/e561e563-f494-44d6-bced-7db92ae72bb9-kube-api-access-wxt9t\") pod \"maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6\" (UID: \"e561e563-f494-44d6-bced-7db92ae72bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6" Nov 02 23:50:13.892282 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:13.892251 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e561e563-f494-44d6-bced-7db92ae72bb9-maas-api-tls\") pod \"maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6\" (UID: \"e561e563-f494-44d6-bced-7db92ae72bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6" Nov 02 23:50:13.894933 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:13.894900 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e561e563-f494-44d6-bced-7db92ae72bb9-maas-api-tls\") pod \"maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6\" (UID: \"e561e563-f494-44d6-bced-7db92ae72bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6" Nov 02 23:50:14.052144 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:14.052097 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6" Nov 02 23:50:14.440925 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:14.440898 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6"] Nov 02 23:50:14.442598 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:50:14.442571 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode561e563_f494_44d6_bced_7db92ae72bb9.slice/crio-e27d9871919c6969ef8be04aef4e99afb02976e60c9582dbf3f3976a2e5c02b0 WatchSource:0}: Error finding container e27d9871919c6969ef8be04aef4e99afb02976e60c9582dbf3f3976a2e5c02b0: Status 404 returned error can't find the container with id e27d9871919c6969ef8be04aef4e99afb02976e60c9582dbf3f3976a2e5c02b0 Nov 02 23:50:15.324415 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:15.324384 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6" event={"ID":"e561e563-f494-44d6-bced-7db92ae72bb9","Type":"ContainerStarted","Data":"e27d9871919c6969ef8be04aef4e99afb02976e60c9582dbf3f3976a2e5c02b0"} Nov 02 23:50:16.333369 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:16.333330 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6" event={"ID":"e561e563-f494-44d6-bced-7db92ae72bb9","Type":"ContainerStarted","Data":"3c780a5e7b533c27534330fbba210f8af8ea40465e26fa7d8b2e4c2ab2003f66"} Nov 02 23:50:16.333827 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:16.333438 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6" Nov 02 23:50:16.450446 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:16.450383 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6" podStartSLOduration=2.6071657310000003 podStartE2EDuration="3.450366499s" totalImagesPullingTime="843.200768ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:50:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:50:16.449275961 +0000 UTC m=+3702.687773648" watchObservedRunningTime="2026-11-02 23:50:16.450366499 +0000 UTC m=+3702.688864162" Nov 02 23:50:22.347547 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:50:22.347514 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6" Nov 02 23:50:27.845772 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:50:27.845734 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:50:57.845121 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:50:57.845086 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:51:27.846731 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:51:27.846694 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:51:47.687735 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:51:47.687188 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6"] Nov 02 23:51:47.688546 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:51:47.688498 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6" podUID="e561e563-f494-44d6-bced-7db92ae72bb9" containerName="maas-api" containerID="cri-o://3c780a5e7b533c27534330fbba210f8af8ea40465e26fa7d8b2e4c2ab2003f66" gracePeriod=30 Nov 02 23:51:47.987748 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:51:47.987719 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6" Nov 02 23:51:48.046013 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:51:48.045981 5727 generic.go:356] "Generic (PLEG): container finished" podID="e561e563-f494-44d6-bced-7db92ae72bb9" containerID="3c780a5e7b533c27534330fbba210f8af8ea40465e26fa7d8b2e4c2ab2003f66" exitCode=0 Nov 02 23:51:48.046234 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:51:48.046044 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6" Nov 02 23:51:48.046234 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:51:48.046060 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6" event={"ID":"e561e563-f494-44d6-bced-7db92ae72bb9","Type":"ContainerDied","Data":"3c780a5e7b533c27534330fbba210f8af8ea40465e26fa7d8b2e4c2ab2003f66"} Nov 02 23:51:48.046234 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:51:48.046113 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6" event={"ID":"e561e563-f494-44d6-bced-7db92ae72bb9","Type":"ContainerDied","Data":"e27d9871919c6969ef8be04aef4e99afb02976e60c9582dbf3f3976a2e5c02b0"} Nov 02 23:51:48.046234 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:51:48.046174 5727 scope.go:122] "RemoveContainer" containerID="3c780a5e7b533c27534330fbba210f8af8ea40465e26fa7d8b2e4c2ab2003f66" Nov 02 23:51:48.057819 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:51:48.057797 5727 scope.go:122] "RemoveContainer" containerID="3c780a5e7b533c27534330fbba210f8af8ea40465e26fa7d8b2e4c2ab2003f66" Nov 02 23:51:48.058102 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:51:48.058084 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c780a5e7b533c27534330fbba210f8af8ea40465e26fa7d8b2e4c2ab2003f66\": container with ID starting with 3c780a5e7b533c27534330fbba210f8af8ea40465e26fa7d8b2e4c2ab2003f66 not found: ID does not exist" containerID="3c780a5e7b533c27534330fbba210f8af8ea40465e26fa7d8b2e4c2ab2003f66" Nov 02 23:51:48.058194 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:51:48.058110 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c780a5e7b533c27534330fbba210f8af8ea40465e26fa7d8b2e4c2ab2003f66"} err="failed to get container status \"3c780a5e7b533c27534330fbba210f8af8ea40465e26fa7d8b2e4c2ab2003f66\": rpc error: code = NotFound desc = could not find container \"3c780a5e7b533c27534330fbba210f8af8ea40465e26fa7d8b2e4c2ab2003f66\": container with ID starting with 3c780a5e7b533c27534330fbba210f8af8ea40465e26fa7d8b2e4c2ab2003f66 not found: ID does not exist" Nov 02 23:51:48.132377 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:51:48.132345 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e561e563-f494-44d6-bced-7db92ae72bb9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e561e563-f494-44d6-bced-7db92ae72bb9-openshift-service-ca\") pod \"e561e563-f494-44d6-bced-7db92ae72bb9\" (UID: \"e561e563-f494-44d6-bced-7db92ae72bb9\") " Nov 02 23:51:48.132556 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:51:48.132490 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e561e563-f494-44d6-bced-7db92ae72bb9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e561e563-f494-44d6-bced-7db92ae72bb9-maas-api-tls\") pod \"e561e563-f494-44d6-bced-7db92ae72bb9\" (UID: \"e561e563-f494-44d6-bced-7db92ae72bb9\") " Nov 02 23:51:48.132556 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:51:48.132541 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e561e563-f494-44d6-bced-7db92ae72bb9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e561e563-f494-44d6-bced-7db92ae72bb9-metrics-certs\") pod \"e561e563-f494-44d6-bced-7db92ae72bb9\" (UID: \"e561e563-f494-44d6-bced-7db92ae72bb9\") " Nov 02 23:51:48.132683 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:51:48.132671 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e561e563-f494-44d6-bced-7db92ae72bb9-kube-api-access-wxt9t\" (UniqueName: \"kubernetes.io/projected/e561e563-f494-44d6-bced-7db92ae72bb9-kube-api-access-wxt9t\") pod \"e561e563-f494-44d6-bced-7db92ae72bb9\" (UID: \"e561e563-f494-44d6-bced-7db92ae72bb9\") " Nov 02 23:51:48.132739 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:51:48.132675 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e561e563-f494-44d6-bced-7db92ae72bb9-openshift-service-ca" pod "e561e563-f494-44d6-bced-7db92ae72bb9" (UID: "e561e563-f494-44d6-bced-7db92ae72bb9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:51:48.132999 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:51:48.132971 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e561e563-f494-44d6-bced-7db92ae72bb9-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:51:48.135589 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:51:48.135523 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e561e563-f494-44d6-bced-7db92ae72bb9-maas-api-tls" pod "e561e563-f494-44d6-bced-7db92ae72bb9" (UID: "e561e563-f494-44d6-bced-7db92ae72bb9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:51:48.135589 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:51:48.135579 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e561e563-f494-44d6-bced-7db92ae72bb9-kube-api-access-wxt9t" pod "e561e563-f494-44d6-bced-7db92ae72bb9" (UID: "e561e563-f494-44d6-bced-7db92ae72bb9"). InnerVolumeSpecName "kube-api-access-wxt9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:51:48.135733 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:51:48.135630 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e561e563-f494-44d6-bced-7db92ae72bb9-metrics-certs" pod "e561e563-f494-44d6-bced-7db92ae72bb9" (UID: "e561e563-f494-44d6-bced-7db92ae72bb9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:51:48.234331 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:51:48.234249 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxt9t\" (UniqueName: \"kubernetes.io/projected/e561e563-f494-44d6-bced-7db92ae72bb9-kube-api-access-wxt9t\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:51:48.234331 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:51:48.234285 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e561e563-f494-44d6-bced-7db92ae72bb9-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:51:48.234331 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:51:48.234298 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e561e563-f494-44d6-bced-7db92ae72bb9-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:51:48.472418 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:51:48.472381 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6"] Nov 02 23:51:48.476796 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:51:48.476769 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-d75a69-57dd8475c9-2vck6"] Nov 02 23:51:49.883975 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:51:49.883940 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e561e563-f494-44d6-bced-7db92ae72bb9" path="/var/lib/kubelet/pods/e561e563-f494-44d6-bced-7db92ae72bb9/volumes" Nov 02 23:51:57.849895 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:51:57.849854 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:52:10.321328 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:52:10.321287 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0f7002-6495d97797-ql9sw"] Nov 02 23:52:10.323843 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:52:10.322302 5727 cpu_manager.go:414] "containerMap: removing container" podUID="e561e563-f494-44d6-bced-7db92ae72bb9" containerName="maas-api" Nov 02 23:52:10.323843 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:52:10.322323 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e561e563-f494-44d6-bced-7db92ae72bb9" containerName="maas-api" Nov 02 23:52:10.323843 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:52:10.322551 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="e561e563-f494-44d6-bced-7db92ae72bb9" containerName="maas-api" Nov 02 23:52:10.327205 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:52:10.327121 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f7002-6495d97797-ql9sw" Nov 02 23:52:10.379958 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:52:10.379927 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-0f7002-dockercfg-xlpjh\"" Nov 02 23:52:10.382248 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:52:10.382223 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0f7002-6495d97797-ql9sw"] Nov 02 23:52:10.409091 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:52:10.409052 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4lx8\" (UniqueName: \"kubernetes.io/projected/e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0-kube-api-access-t4lx8\") pod \"payload-processing-e2e-extmod-a-0f7002-6495d97797-ql9sw\" (UID: \"e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0f7002-6495d97797-ql9sw" Nov 02 23:52:10.409301 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:52:10.409228 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0f7002-6495d97797-ql9sw\" (UID: \"e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0f7002-6495d97797-ql9sw" Nov 02 23:52:10.426499 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:52:10.426464 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-0f7002\"" Nov 02 23:52:10.510659 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:52:10.510618 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4lx8\" (UniqueName: \"kubernetes.io/projected/e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0-kube-api-access-t4lx8\") pod \"payload-processing-e2e-extmod-a-0f7002-6495d97797-ql9sw\" (UID: \"e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0f7002-6495d97797-ql9sw" Nov 02 23:52:10.510863 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:52:10.510756 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0f7002-6495d97797-ql9sw\" (UID: \"e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0f7002-6495d97797-ql9sw" Nov 02 23:52:10.511713 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:52:10.511687 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0f7002-6495d97797-ql9sw\" (UID: \"e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0f7002-6495d97797-ql9sw" Nov 02 23:52:10.575932 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:52:10.575854 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4lx8\" (UniqueName: \"kubernetes.io/projected/e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0-kube-api-access-t4lx8\") pod \"payload-processing-e2e-extmod-a-0f7002-6495d97797-ql9sw\" (UID: \"e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0f7002-6495d97797-ql9sw" Nov 02 23:52:10.653495 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:52:10.653456 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f7002-6495d97797-ql9sw" Nov 02 23:52:11.035317 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:52:11.035290 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0f7002-6495d97797-ql9sw"] Nov 02 23:52:11.037005 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:52:11.036976 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode84b19b9_9f74_4d5d_ab9f_dbbefd6d06f0.slice/crio-2a7375559ed2213c467b21cffa197bda651258f9aa950a980fafec59ca91a54b WatchSource:0}: Error finding container 2a7375559ed2213c467b21cffa197bda651258f9aa950a980fafec59ca91a54b: Status 404 returned error can't find the container with id 2a7375559ed2213c467b21cffa197bda651258f9aa950a980fafec59ca91a54b Nov 02 23:52:11.236925 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:52:11.236887 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f7002-6495d97797-ql9sw" event={"ID":"e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0","Type":"ContainerStarted","Data":"a12ef48c5efcf05fee561f4066f54d6fda6c87c7cf4eabb975dcbccfc3d25af8"} Nov 02 23:52:11.237161 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:52:11.236942 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f7002-6495d97797-ql9sw" event={"ID":"e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0","Type":"ContainerStarted","Data":"2a7375559ed2213c467b21cffa197bda651258f9aa950a980fafec59ca91a54b"} Nov 02 23:52:11.237161 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:52:11.237118 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f7002-6495d97797-ql9sw" Nov 02 23:52:11.357729 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:52:11.357666 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f7002-6495d97797-ql9sw" podStartSLOduration=1.357651983 podStartE2EDuration="1.357651983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:52:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:52:11.353767201 +0000 UTC m=+3817.592264874" watchObservedRunningTime="2026-11-02 23:52:11.357651983 +0000 UTC m=+3817.596149649" Nov 02 23:52:22.249015 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:52:22.248985 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f7002-6495d97797-ql9sw" Nov 02 23:52:27.847341 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:52:27.847304 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:52:57.850000 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:52:57.849955 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:53:01.995345 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:01.995303 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g"] Nov 02 23:53:01.995876 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:01.995549 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g" podUID="20a75303-a945-49f6-9e10-1d9d5b25e6ea" containerName="maas-api" containerID="cri-o://08a217c6d2732d3706839d5a89530cc4791d82957e26ab5b6ed67dc691f01a81" gracePeriod=30 Nov 02 23:53:02.284607 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:02.284577 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g" Nov 02 23:53:02.337494 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:02.337456 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20a75303-a945-49f6-9e10-1d9d5b25e6ea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/20a75303-a945-49f6-9e10-1d9d5b25e6ea-metrics-certs\") pod \"20a75303-a945-49f6-9e10-1d9d5b25e6ea\" (UID: \"20a75303-a945-49f6-9e10-1d9d5b25e6ea\") " Nov 02 23:53:02.337694 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:02.337521 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20a75303-a945-49f6-9e10-1d9d5b25e6ea-kube-api-access-gpdbg\" (UniqueName: \"kubernetes.io/projected/20a75303-a945-49f6-9e10-1d9d5b25e6ea-kube-api-access-gpdbg\") pod \"20a75303-a945-49f6-9e10-1d9d5b25e6ea\" (UID: \"20a75303-a945-49f6-9e10-1d9d5b25e6ea\") " Nov 02 23:53:02.337694 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:02.337555 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20a75303-a945-49f6-9e10-1d9d5b25e6ea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a75303-a945-49f6-9e10-1d9d5b25e6ea-maas-api-tls\") pod \"20a75303-a945-49f6-9e10-1d9d5b25e6ea\" (UID: \"20a75303-a945-49f6-9e10-1d9d5b25e6ea\") " Nov 02 23:53:02.337849 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:02.337760 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20a75303-a945-49f6-9e10-1d9d5b25e6ea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a75303-a945-49f6-9e10-1d9d5b25e6ea-openshift-service-ca\") pod \"20a75303-a945-49f6-9e10-1d9d5b25e6ea\" (UID: \"20a75303-a945-49f6-9e10-1d9d5b25e6ea\") " Nov 02 23:53:02.338194 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:02.338119 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a75303-a945-49f6-9e10-1d9d5b25e6ea-openshift-service-ca" pod "20a75303-a945-49f6-9e10-1d9d5b25e6ea" (UID: "20a75303-a945-49f6-9e10-1d9d5b25e6ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:53:02.338456 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:02.338437 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a75303-a945-49f6-9e10-1d9d5b25e6ea-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:02.339886 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:02.339860 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a75303-a945-49f6-9e10-1d9d5b25e6ea-metrics-certs" pod "20a75303-a945-49f6-9e10-1d9d5b25e6ea" (UID: "20a75303-a945-49f6-9e10-1d9d5b25e6ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:53:02.339886 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:02.339876 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a75303-a945-49f6-9e10-1d9d5b25e6ea-maas-api-tls" pod "20a75303-a945-49f6-9e10-1d9d5b25e6ea" (UID: "20a75303-a945-49f6-9e10-1d9d5b25e6ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:53:02.340059 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:02.339937 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a75303-a945-49f6-9e10-1d9d5b25e6ea-kube-api-access-gpdbg" pod "20a75303-a945-49f6-9e10-1d9d5b25e6ea" (UID: "20a75303-a945-49f6-9e10-1d9d5b25e6ea"). InnerVolumeSpecName "kube-api-access-gpdbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:53:02.439901 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:02.439860 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20a75303-a945-49f6-9e10-1d9d5b25e6ea-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:02.439901 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:02.439900 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpdbg\" (UniqueName: \"kubernetes.io/projected/20a75303-a945-49f6-9e10-1d9d5b25e6ea-kube-api-access-gpdbg\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:02.440118 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:02.439916 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a75303-a945-49f6-9e10-1d9d5b25e6ea-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:02.636358 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:02.636323 5727 generic.go:356] "Generic (PLEG): container finished" podID="20a75303-a945-49f6-9e10-1d9d5b25e6ea" containerID="08a217c6d2732d3706839d5a89530cc4791d82957e26ab5b6ed67dc691f01a81" exitCode=0 Nov 02 23:53:02.636569 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:02.636368 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g" Nov 02 23:53:02.636569 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:02.636407 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g" event={"ID":"20a75303-a945-49f6-9e10-1d9d5b25e6ea","Type":"ContainerDied","Data":"08a217c6d2732d3706839d5a89530cc4791d82957e26ab5b6ed67dc691f01a81"} Nov 02 23:53:02.636569 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:02.636465 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g" event={"ID":"20a75303-a945-49f6-9e10-1d9d5b25e6ea","Type":"ContainerDied","Data":"8dc39a9dfdb32771bbf2ed44e9cbaa24ecfa4769b4de53f5a3404b036605b15f"} Nov 02 23:53:02.636569 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:02.636486 5727 scope.go:122] "RemoveContainer" containerID="08a217c6d2732d3706839d5a89530cc4791d82957e26ab5b6ed67dc691f01a81" Nov 02 23:53:02.650542 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:02.649296 5727 scope.go:122] "RemoveContainer" containerID="08a217c6d2732d3706839d5a89530cc4791d82957e26ab5b6ed67dc691f01a81" Nov 02 23:53:02.650542 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:53:02.649620 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08a217c6d2732d3706839d5a89530cc4791d82957e26ab5b6ed67dc691f01a81\": container with ID starting with 08a217c6d2732d3706839d5a89530cc4791d82957e26ab5b6ed67dc691f01a81 not found: ID does not exist" containerID="08a217c6d2732d3706839d5a89530cc4791d82957e26ab5b6ed67dc691f01a81" Nov 02 23:53:02.650542 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:02.649649 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08a217c6d2732d3706839d5a89530cc4791d82957e26ab5b6ed67dc691f01a81"} err="failed to get container status \"08a217c6d2732d3706839d5a89530cc4791d82957e26ab5b6ed67dc691f01a81\": rpc error: code = NotFound desc = could not find container \"08a217c6d2732d3706839d5a89530cc4791d82957e26ab5b6ed67dc691f01a81\": container with ID starting with 08a217c6d2732d3706839d5a89530cc4791d82957e26ab5b6ed67dc691f01a81 not found: ID does not exist" Nov 02 23:53:02.802873 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:02.802816 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g"] Nov 02 23:53:02.852053 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:02.852008 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3058ee-57c9fb46dd-h8k6g"] Nov 02 23:53:03.883913 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:03.883877 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20a75303-a945-49f6-9e10-1d9d5b25e6ea" path="/var/lib/kubelet/pods/20a75303-a945-49f6-9e10-1d9d5b25e6ea/volumes" Nov 02 23:53:15.494260 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:15.494165 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-942137-6d5cc6b689-wjbdr"] Nov 02 23:53:15.603202 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:15.603160 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-942137-5c7c58df55-7r9vb"] Nov 02 23:53:20.494982 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:20.494937 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-942137-6d5cc6b689-wjbdr" podUID="eb5397a3-06aa-4c5e-ac18-9b3411a753d0" containerName="payload-processing" containerID="cri-o://8c8f45c74d3711a5d7531cdb0a19708ff11076d33dd139021e2e7661e506d591" gracePeriod=25 Nov 02 23:53:20.604350 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:20.604282 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-942137-5c7c58df55-7r9vb" podUID="a5ad8962-82c6-4113-933e-9d295480223d" containerName="payload-pre-processing" containerID="cri-o://cb2e457a6622134b08d9b5ccd67026da9ea9e57113f4770187105b76e4fd7662" gracePeriod=25 Nov 02 23:53:20.779545 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:20.779510 5727 generic.go:356] "Generic (PLEG): container finished" podID="a5ad8962-82c6-4113-933e-9d295480223d" containerID="cb2e457a6622134b08d9b5ccd67026da9ea9e57113f4770187105b76e4fd7662" exitCode=0 Nov 02 23:53:20.779736 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:20.779702 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-942137-5c7c58df55-7r9vb" event={"ID":"a5ad8962-82c6-4113-933e-9d295480223d","Type":"ContainerDied","Data":"cb2e457a6622134b08d9b5ccd67026da9ea9e57113f4770187105b76e4fd7662"} Nov 02 23:53:20.781184 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:20.781159 5727 generic.go:356] "Generic (PLEG): container finished" podID="eb5397a3-06aa-4c5e-ac18-9b3411a753d0" containerID="8c8f45c74d3711a5d7531cdb0a19708ff11076d33dd139021e2e7661e506d591" exitCode=0 Nov 02 23:53:20.781340 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:20.781315 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-942137-6d5cc6b689-wjbdr" event={"ID":"eb5397a3-06aa-4c5e-ac18-9b3411a753d0","Type":"ContainerDied","Data":"8c8f45c74d3711a5d7531cdb0a19708ff11076d33dd139021e2e7661e506d591"} Nov 02 23:53:20.781407 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:20.781363 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-942137-6d5cc6b689-wjbdr" event={"ID":"eb5397a3-06aa-4c5e-ac18-9b3411a753d0","Type":"ContainerDied","Data":"70d16fca209bc8e5224d9a6bd5404f2e02f2ace6d5b2f6f0fd16b07b0fa87355"} Nov 02 23:53:20.781407 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:20.781379 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70d16fca209bc8e5224d9a6bd5404f2e02f2ace6d5b2f6f0fd16b07b0fa87355" Nov 02 23:53:20.798745 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:20.798375 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-942137-6d5cc6b689-wjbdr" Nov 02 23:53:20.859657 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:20.859626 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb5397a3-06aa-4c5e-ac18-9b3411a753d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb5397a3-06aa-4c5e-ac18-9b3411a753d0-plugins-config-volume\") pod \"eb5397a3-06aa-4c5e-ac18-9b3411a753d0\" (UID: \"eb5397a3-06aa-4c5e-ac18-9b3411a753d0\") " Nov 02 23:53:20.859868 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:20.859848 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb5397a3-06aa-4c5e-ac18-9b3411a753d0-kube-api-access-6b459\" (UniqueName: \"kubernetes.io/projected/eb5397a3-06aa-4c5e-ac18-9b3411a753d0-kube-api-access-6b459\") pod \"eb5397a3-06aa-4c5e-ac18-9b3411a753d0\" (UID: \"eb5397a3-06aa-4c5e-ac18-9b3411a753d0\") " Nov 02 23:53:20.860502 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:20.860458 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb5397a3-06aa-4c5e-ac18-9b3411a753d0-plugins-config-volume" pod "eb5397a3-06aa-4c5e-ac18-9b3411a753d0" (UID: "eb5397a3-06aa-4c5e-ac18-9b3411a753d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:53:20.862954 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:20.862926 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb5397a3-06aa-4c5e-ac18-9b3411a753d0-kube-api-access-6b459" pod "eb5397a3-06aa-4c5e-ac18-9b3411a753d0" (UID: "eb5397a3-06aa-4c5e-ac18-9b3411a753d0"). InnerVolumeSpecName "kube-api-access-6b459". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:53:20.887762 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:20.887735 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-942137-5c7c58df55-7r9vb" Nov 02 23:53:20.961089 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:20.961057 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5ad8962-82c6-4113-933e-9d295480223d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5ad8962-82c6-4113-933e-9d295480223d-plugins-config-volume\") pod \"a5ad8962-82c6-4113-933e-9d295480223d\" (UID: \"a5ad8962-82c6-4113-933e-9d295480223d\") " Nov 02 23:53:20.961327 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:20.961236 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5ad8962-82c6-4113-933e-9d295480223d-kube-api-access-ts5k2\" (UniqueName: \"kubernetes.io/projected/a5ad8962-82c6-4113-933e-9d295480223d-kube-api-access-ts5k2\") pod \"a5ad8962-82c6-4113-933e-9d295480223d\" (UID: \"a5ad8962-82c6-4113-933e-9d295480223d\") " Nov 02 23:53:20.961613 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:20.961583 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5ad8962-82c6-4113-933e-9d295480223d-plugins-config-volume" pod "a5ad8962-82c6-4113-933e-9d295480223d" (UID: "a5ad8962-82c6-4113-933e-9d295480223d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:53:20.961695 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:20.961672 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b459\" (UniqueName: \"kubernetes.io/projected/eb5397a3-06aa-4c5e-ac18-9b3411a753d0-kube-api-access-6b459\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:20.961750 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:20.961696 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb5397a3-06aa-4c5e-ac18-9b3411a753d0-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:20.963635 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:20.963611 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ad8962-82c6-4113-933e-9d295480223d-kube-api-access-ts5k2" pod "a5ad8962-82c6-4113-933e-9d295480223d" (UID: "a5ad8962-82c6-4113-933e-9d295480223d"). InnerVolumeSpecName "kube-api-access-ts5k2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:53:21.062858 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:21.062809 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5ad8962-82c6-4113-933e-9d295480223d-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:21.062858 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:21.062850 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts5k2\" (UniqueName: \"kubernetes.io/projected/a5ad8962-82c6-4113-933e-9d295480223d-kube-api-access-ts5k2\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:21.790992 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:21.790895 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-942137-5c7c58df55-7r9vb" Nov 02 23:53:21.790992 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:21.790916 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-942137-5c7c58df55-7r9vb" event={"ID":"a5ad8962-82c6-4113-933e-9d295480223d","Type":"ContainerDied","Data":"343c6bdc7ff42f2f91854c0f85c848ba8bc6216083219dfd18487cb67702fbf7"} Nov 02 23:53:21.790992 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:21.790895 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-942137-6d5cc6b689-wjbdr" Nov 02 23:53:21.791631 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:21.791031 5727 scope.go:122] "RemoveContainer" containerID="cb2e457a6622134b08d9b5ccd67026da9ea9e57113f4770187105b76e4fd7662" Nov 02 23:53:22.007004 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:22.006968 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-942137-5c7c58df55-7r9vb"] Nov 02 23:53:22.012376 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:22.012350 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-942137-5c7c58df55-7r9vb"] Nov 02 23:53:22.166305 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:22.166270 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-942137-6d5cc6b689-wjbdr"] Nov 02 23:53:22.171592 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:22.171558 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-942137-6d5cc6b689-wjbdr"] Nov 02 23:53:23.885604 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:23.885565 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5ad8962-82c6-4113-933e-9d295480223d" path="/var/lib/kubelet/pods/a5ad8962-82c6-4113-933e-9d295480223d/volumes" Nov 02 23:53:23.886072 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:23.886054 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb5397a3-06aa-4c5e-ac18-9b3411a753d0" path="/var/lib/kubelet/pods/eb5397a3-06aa-4c5e-ac18-9b3411a753d0/volumes" Nov 02 23:53:27.849281 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:53:27.849239 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:53:28.940680 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:28.940640 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg"] Nov 02 23:53:28.941245 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:28.940871 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg" podUID="1986d8d6-894f-49e7-a357-24fff971f75b" containerName="maas-api" containerID="cri-o://1c89cc0fc40ff9c1fdf63ce9a7f925c568528d2ac7d622ebe456e5fe3c67d2f6" gracePeriod=30 Nov 02 23:53:29.245386 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:29.245358 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg" Nov 02 23:53:29.377214 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:29.377182 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1986d8d6-894f-49e7-a357-24fff971f75b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1986d8d6-894f-49e7-a357-24fff971f75b-maas-api-tls\") pod \"1986d8d6-894f-49e7-a357-24fff971f75b\" (UID: \"1986d8d6-894f-49e7-a357-24fff971f75b\") " Nov 02 23:53:29.377420 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:29.377348 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1986d8d6-894f-49e7-a357-24fff971f75b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1986d8d6-894f-49e7-a357-24fff971f75b-openshift-service-ca\") pod \"1986d8d6-894f-49e7-a357-24fff971f75b\" (UID: \"1986d8d6-894f-49e7-a357-24fff971f75b\") " Nov 02 23:53:29.377499 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:29.377483 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1986d8d6-894f-49e7-a357-24fff971f75b-kube-api-access-fzjt4\" (UniqueName: \"kubernetes.io/projected/1986d8d6-894f-49e7-a357-24fff971f75b-kube-api-access-fzjt4\") pod \"1986d8d6-894f-49e7-a357-24fff971f75b\" (UID: \"1986d8d6-894f-49e7-a357-24fff971f75b\") " Nov 02 23:53:29.377567 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:29.377528 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1986d8d6-894f-49e7-a357-24fff971f75b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1986d8d6-894f-49e7-a357-24fff971f75b-metrics-certs\") pod \"1986d8d6-894f-49e7-a357-24fff971f75b\" (UID: \"1986d8d6-894f-49e7-a357-24fff971f75b\") " Nov 02 23:53:29.378146 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:29.378100 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1986d8d6-894f-49e7-a357-24fff971f75b-openshift-service-ca" pod "1986d8d6-894f-49e7-a357-24fff971f75b" (UID: "1986d8d6-894f-49e7-a357-24fff971f75b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:53:29.380309 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:29.380275 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1986d8d6-894f-49e7-a357-24fff971f75b-metrics-certs" pod "1986d8d6-894f-49e7-a357-24fff971f75b" (UID: "1986d8d6-894f-49e7-a357-24fff971f75b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:53:29.380552 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:29.380525 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1986d8d6-894f-49e7-a357-24fff971f75b-maas-api-tls" pod "1986d8d6-894f-49e7-a357-24fff971f75b" (UID: "1986d8d6-894f-49e7-a357-24fff971f75b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:53:29.380737 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:29.380711 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1986d8d6-894f-49e7-a357-24fff971f75b-kube-api-access-fzjt4" pod "1986d8d6-894f-49e7-a357-24fff971f75b" (UID: "1986d8d6-894f-49e7-a357-24fff971f75b"). InnerVolumeSpecName "kube-api-access-fzjt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:53:29.479018 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:29.478930 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzjt4\" (UniqueName: \"kubernetes.io/projected/1986d8d6-894f-49e7-a357-24fff971f75b-kube-api-access-fzjt4\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:29.479018 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:29.478969 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1986d8d6-894f-49e7-a357-24fff971f75b-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:29.479018 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:29.478979 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1986d8d6-894f-49e7-a357-24fff971f75b-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:29.479018 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:29.478988 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1986d8d6-894f-49e7-a357-24fff971f75b-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:29.863943 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:29.863905 5727 generic.go:356] "Generic (PLEG): container finished" podID="1986d8d6-894f-49e7-a357-24fff971f75b" containerID="1c89cc0fc40ff9c1fdf63ce9a7f925c568528d2ac7d622ebe456e5fe3c67d2f6" exitCode=0 Nov 02 23:53:29.864173 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:29.863959 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg" Nov 02 23:53:29.864173 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:29.864007 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg" event={"ID":"1986d8d6-894f-49e7-a357-24fff971f75b","Type":"ContainerDied","Data":"1c89cc0fc40ff9c1fdf63ce9a7f925c568528d2ac7d622ebe456e5fe3c67d2f6"} Nov 02 23:53:29.864173 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:29.864080 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg" event={"ID":"1986d8d6-894f-49e7-a357-24fff971f75b","Type":"ContainerDied","Data":"80a75cd3abe329118889b234b535beb1260848c3bd8217a53115ef00aa4c822f"} Nov 02 23:53:29.864173 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:29.864114 5727 scope.go:122] "RemoveContainer" containerID="1c89cc0fc40ff9c1fdf63ce9a7f925c568528d2ac7d622ebe456e5fe3c67d2f6" Nov 02 23:53:29.892691 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:29.892667 5727 scope.go:122] "RemoveContainer" containerID="1c89cc0fc40ff9c1fdf63ce9a7f925c568528d2ac7d622ebe456e5fe3c67d2f6" Nov 02 23:53:29.893192 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:53:29.893163 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c89cc0fc40ff9c1fdf63ce9a7f925c568528d2ac7d622ebe456e5fe3c67d2f6\": container with ID starting with 1c89cc0fc40ff9c1fdf63ce9a7f925c568528d2ac7d622ebe456e5fe3c67d2f6 not found: ID does not exist" containerID="1c89cc0fc40ff9c1fdf63ce9a7f925c568528d2ac7d622ebe456e5fe3c67d2f6" Nov 02 23:53:29.893302 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:29.893200 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c89cc0fc40ff9c1fdf63ce9a7f925c568528d2ac7d622ebe456e5fe3c67d2f6"} err="failed to get container status \"1c89cc0fc40ff9c1fdf63ce9a7f925c568528d2ac7d622ebe456e5fe3c67d2f6\": rpc error: code = NotFound desc = could not find container \"1c89cc0fc40ff9c1fdf63ce9a7f925c568528d2ac7d622ebe456e5fe3c67d2f6\": container with ID starting with 1c89cc0fc40ff9c1fdf63ce9a7f925c568528d2ac7d622ebe456e5fe3c67d2f6 not found: ID does not exist" Nov 02 23:53:30.047699 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:30.047657 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg"] Nov 02 23:53:30.052325 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:30.052298 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1498df-6977cf4c86-hbmfg"] Nov 02 23:53:30.507820 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:30.507784 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8"] Nov 02 23:53:30.508506 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:30.508483 5727 cpu_manager.go:414] "containerMap: removing container" podUID="eb5397a3-06aa-4c5e-ac18-9b3411a753d0" containerName="payload-processing" Nov 02 23:53:30.508506 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:30.508503 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb5397a3-06aa-4c5e-ac18-9b3411a753d0" containerName="payload-processing" Nov 02 23:53:30.508728 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:30.508527 5727 cpu_manager.go:414] "containerMap: removing container" podUID="20a75303-a945-49f6-9e10-1d9d5b25e6ea" containerName="maas-api" Nov 02 23:53:30.508728 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:30.508532 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20a75303-a945-49f6-9e10-1d9d5b25e6ea" containerName="maas-api" Nov 02 23:53:30.508728 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:30.508541 5727 cpu_manager.go:414] "containerMap: removing container" podUID="1986d8d6-894f-49e7-a357-24fff971f75b" containerName="maas-api" Nov 02 23:53:30.508728 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:30.508547 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1986d8d6-894f-49e7-a357-24fff971f75b" containerName="maas-api" Nov 02 23:53:30.508728 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:30.508566 5727 cpu_manager.go:414] "containerMap: removing container" podUID="a5ad8962-82c6-4113-933e-9d295480223d" containerName="payload-pre-processing" Nov 02 23:53:30.508728 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:30.508572 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5ad8962-82c6-4113-933e-9d295480223d" containerName="payload-pre-processing" Nov 02 23:53:30.508728 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:30.508689 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb5397a3-06aa-4c5e-ac18-9b3411a753d0" containerName="payload-processing" Nov 02 23:53:30.508728 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:30.508699 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="20a75303-a945-49f6-9e10-1d9d5b25e6ea" containerName="maas-api" Nov 02 23:53:30.508728 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:30.508704 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="1986d8d6-894f-49e7-a357-24fff971f75b" containerName="maas-api" Nov 02 23:53:30.508728 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:30.508713 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5ad8962-82c6-4113-933e-9d295480223d" containerName="payload-pre-processing" Nov 02 23:53:30.513706 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:30.513681 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8" Nov 02 23:53:30.566348 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:30.566315 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-0f7002-serving-cert\"" Nov 02 23:53:30.570972 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:30.570933 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8"] Nov 02 23:53:30.597266 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:30.597228 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/097f0270-0698-4864-b784-b702e76a7159-metrics-certs\") pod \"maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8\" (UID: \"097f0270-0698-4864-b784-b702e76a7159\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8" Nov 02 23:53:30.597460 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:30.597272 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/097f0270-0698-4864-b784-b702e76a7159-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8\" (UID: \"097f0270-0698-4864-b784-b702e76a7159\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8" Nov 02 23:53:30.597460 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:30.597420 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg5wf\" (UniqueName: \"kubernetes.io/projected/097f0270-0698-4864-b784-b702e76a7159-kube-api-access-tg5wf\") pod \"maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8\" (UID: \"097f0270-0698-4864-b784-b702e76a7159\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8" Nov 02 23:53:30.597639 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:30.597611 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/097f0270-0698-4864-b784-b702e76a7159-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8\" (UID: \"097f0270-0698-4864-b784-b702e76a7159\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8" Nov 02 23:53:30.698763 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:30.698727 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/097f0270-0698-4864-b784-b702e76a7159-metrics-certs\") pod \"maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8\" (UID: \"097f0270-0698-4864-b784-b702e76a7159\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8" Nov 02 23:53:30.698973 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:30.698768 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/097f0270-0698-4864-b784-b702e76a7159-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8\" (UID: \"097f0270-0698-4864-b784-b702e76a7159\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8" Nov 02 23:53:30.698973 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:30.698848 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg5wf\" (UniqueName: \"kubernetes.io/projected/097f0270-0698-4864-b784-b702e76a7159-kube-api-access-tg5wf\") pod \"maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8\" (UID: \"097f0270-0698-4864-b784-b702e76a7159\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8" Nov 02 23:53:30.699110 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:53:30.698967 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f7002-serving-cert: secret "maas-api-e2e-extmod-a-0f7002-serving-cert" not found Nov 02 23:53:30.699110 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:30.698972 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/097f0270-0698-4864-b784-b702e76a7159-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8\" (UID: \"097f0270-0698-4864-b784-b702e76a7159\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8" Nov 02 23:53:30.699110 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:53:30.699044 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/097f0270-0698-4864-b784-b702e76a7159-maas-api-tls podName:097f0270-0698-4864-b784-b702e76a7159 nodeName:}" failed. No retries permitted until 2026-11-02 23:53:31.199023767 +0000 UTC m=+3897.437521421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/097f0270-0698-4864-b784-b702e76a7159-maas-api-tls") pod "maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8" (UID: "097f0270-0698-4864-b784-b702e76a7159") : secret "maas-api-e2e-extmod-a-0f7002-serving-cert" not found Nov 02 23:53:30.699632 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:30.699608 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/097f0270-0698-4864-b784-b702e76a7159-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8\" (UID: \"097f0270-0698-4864-b784-b702e76a7159\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8" Nov 02 23:53:30.702412 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:30.702377 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/097f0270-0698-4864-b784-b702e76a7159-metrics-certs\") pod \"maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8\" (UID: \"097f0270-0698-4864-b784-b702e76a7159\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8" Nov 02 23:53:30.766572 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:30.766482 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg5wf\" (UniqueName: \"kubernetes.io/projected/097f0270-0698-4864-b784-b702e76a7159-kube-api-access-tg5wf\") pod \"maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8\" (UID: \"097f0270-0698-4864-b784-b702e76a7159\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8" Nov 02 23:53:31.207016 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:31.206978 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/097f0270-0698-4864-b784-b702e76a7159-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8\" (UID: \"097f0270-0698-4864-b784-b702e76a7159\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8" Nov 02 23:53:31.209828 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:31.209793 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/097f0270-0698-4864-b784-b702e76a7159-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8\" (UID: \"097f0270-0698-4864-b784-b702e76a7159\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8" Nov 02 23:53:31.429549 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:31.429511 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8" Nov 02 23:53:31.818918 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:31.818890 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8"] Nov 02 23:53:31.821264 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:53:31.821231 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod097f0270_0698_4864_b784_b702e76a7159.slice/crio-eb4e6bd93bd8e13b73d7b237b4665fe2088c24008987e39ec38bd0866c2536bf WatchSource:0}: Error finding container eb4e6bd93bd8e13b73d7b237b4665fe2088c24008987e39ec38bd0866c2536bf: Status 404 returned error can't find the container with id eb4e6bd93bd8e13b73d7b237b4665fe2088c24008987e39ec38bd0866c2536bf Nov 02 23:53:31.823022 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:31.823000 5727 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 23:53:31.886780 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:31.886744 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1986d8d6-894f-49e7-a357-24fff971f75b" path="/var/lib/kubelet/pods/1986d8d6-894f-49e7-a357-24fff971f75b/volumes" Nov 02 23:53:31.887099 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:31.887082 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8" event={"ID":"097f0270-0698-4864-b784-b702e76a7159","Type":"ContainerStarted","Data":"eb4e6bd93bd8e13b73d7b237b4665fe2088c24008987e39ec38bd0866c2536bf"} Nov 02 23:53:32.896576 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:32.896531 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8" event={"ID":"097f0270-0698-4864-b784-b702e76a7159","Type":"ContainerStarted","Data":"0746dab54d71f24875fdf5bad6d646edac98ca7cb3e4f783393f6060053dccef"} Nov 02 23:53:32.897094 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:32.896606 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8" Nov 02 23:53:33.013513 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:33.013393 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8" podStartSLOduration=2.136559645 podStartE2EDuration="3.013372191s" totalImagesPullingTime="876.812546ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:53:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:53:33.012443275 +0000 UTC m=+3899.250940938" watchObservedRunningTime="2026-11-02 23:53:33.013372191 +0000 UTC m=+3899.251869854" Nov 02 23:53:34.785267 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:34.785232 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 02 23:53:34.801225 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:34.801179 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 23:53:34.870537 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:34.870507 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 02 23:53:34.895826 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:34.895795 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 23:53:38.928352 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:53:38.928324 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8" Nov 02 23:53:57.846552 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:53:57.846510 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:54:08.082062 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:08.082029 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-68467d9546-dzddb"] Nov 02 23:54:08.087483 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:08.087465 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68467d9546-dzddb" Nov 02 23:54:08.120463 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:08.120438 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65550239-10be-4c4e-b1fb-b7e7235bdb49-plugins-config-volume\") pod \"payload-processing-external-model-68467d9546-dzddb\" (UID: \"65550239-10be-4c4e-b1fb-b7e7235bdb49\") " pod="models-as-a-service/payload-processing-external-model-68467d9546-dzddb" Nov 02 23:54:08.120614 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:08.120593 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvcgc\" (UniqueName: \"kubernetes.io/projected/65550239-10be-4c4e-b1fb-b7e7235bdb49-kube-api-access-nvcgc\") pod \"payload-processing-external-model-68467d9546-dzddb\" (UID: \"65550239-10be-4c4e-b1fb-b7e7235bdb49\") " pod="models-as-a-service/payload-processing-external-model-68467d9546-dzddb" Nov 02 23:54:08.120672 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:08.120637 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/65550239-10be-4c4e-b1fb-b7e7235bdb49-routing-overlay-volume\") pod \"payload-processing-external-model-68467d9546-dzddb\" (UID: \"65550239-10be-4c4e-b1fb-b7e7235bdb49\") " pod="models-as-a-service/payload-processing-external-model-68467d9546-dzddb" Nov 02 23:54:08.120753 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:08.120734 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/65550239-10be-4c4e-b1fb-b7e7235bdb49-provider-credentials\") pod \"payload-processing-external-model-68467d9546-dzddb\" (UID: \"65550239-10be-4c4e-b1fb-b7e7235bdb49\") " pod="models-as-a-service/payload-processing-external-model-68467d9546-dzddb" Nov 02 23:54:08.143410 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:08.143389 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-68467d9546-dzddb"] Nov 02 23:54:08.188919 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:08.188892 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 23:54:08.188919 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:08.188913 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 23:54:08.189160 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:08.188897 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-58c6f06e6850\"" Nov 02 23:54:08.189160 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:08.188972 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 23:54:08.189160 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:08.188998 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 23:54:08.189160 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:08.189003 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-d9l5p\"" Nov 02 23:54:08.221836 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:08.221812 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/65550239-10be-4c4e-b1fb-b7e7235bdb49-provider-credentials\") pod \"payload-processing-external-model-68467d9546-dzddb\" (UID: \"65550239-10be-4c4e-b1fb-b7e7235bdb49\") " pod="models-as-a-service/payload-processing-external-model-68467d9546-dzddb" Nov 02 23:54:08.221947 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:08.221899 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65550239-10be-4c4e-b1fb-b7e7235bdb49-plugins-config-volume\") pod \"payload-processing-external-model-68467d9546-dzddb\" (UID: \"65550239-10be-4c4e-b1fb-b7e7235bdb49\") " pod="models-as-a-service/payload-processing-external-model-68467d9546-dzddb" Nov 02 23:54:08.222042 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:08.222023 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvcgc\" (UniqueName: \"kubernetes.io/projected/65550239-10be-4c4e-b1fb-b7e7235bdb49-kube-api-access-nvcgc\") pod \"payload-processing-external-model-68467d9546-dzddb\" (UID: \"65550239-10be-4c4e-b1fb-b7e7235bdb49\") " pod="models-as-a-service/payload-processing-external-model-68467d9546-dzddb" Nov 02 23:54:08.222098 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:08.222075 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/65550239-10be-4c4e-b1fb-b7e7235bdb49-routing-overlay-volume\") pod \"payload-processing-external-model-68467d9546-dzddb\" (UID: \"65550239-10be-4c4e-b1fb-b7e7235bdb49\") " pod="models-as-a-service/payload-processing-external-model-68467d9546-dzddb" Nov 02 23:54:08.222771 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:08.222746 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/65550239-10be-4c4e-b1fb-b7e7235bdb49-routing-overlay-volume\") pod \"payload-processing-external-model-68467d9546-dzddb\" (UID: \"65550239-10be-4c4e-b1fb-b7e7235bdb49\") " pod="models-as-a-service/payload-processing-external-model-68467d9546-dzddb" Nov 02 23:54:08.223009 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:08.222989 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65550239-10be-4c4e-b1fb-b7e7235bdb49-plugins-config-volume\") pod \"payload-processing-external-model-68467d9546-dzddb\" (UID: \"65550239-10be-4c4e-b1fb-b7e7235bdb49\") " pod="models-as-a-service/payload-processing-external-model-68467d9546-dzddb" Nov 02 23:54:08.224590 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:08.224571 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/65550239-10be-4c4e-b1fb-b7e7235bdb49-provider-credentials\") pod \"payload-processing-external-model-68467d9546-dzddb\" (UID: \"65550239-10be-4c4e-b1fb-b7e7235bdb49\") " pod="models-as-a-service/payload-processing-external-model-68467d9546-dzddb" Nov 02 23:54:08.282576 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:08.282552 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvcgc\" (UniqueName: \"kubernetes.io/projected/65550239-10be-4c4e-b1fb-b7e7235bdb49-kube-api-access-nvcgc\") pod \"payload-processing-external-model-68467d9546-dzddb\" (UID: \"65550239-10be-4c4e-b1fb-b7e7235bdb49\") " pod="models-as-a-service/payload-processing-external-model-68467d9546-dzddb" Nov 02 23:54:08.402371 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:08.402289 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68467d9546-dzddb" Nov 02 23:54:08.784961 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:08.784935 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-68467d9546-dzddb"] Nov 02 23:54:08.786375 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:54:08.786346 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65550239_10be_4c4e_b1fb_b7e7235bdb49.slice/crio-2448ac9953edde68c3d5b1415b0d3af6ab3a755afbb8ae03ab27edc90d8935d5 WatchSource:0}: Error finding container 2448ac9953edde68c3d5b1415b0d3af6ab3a755afbb8ae03ab27edc90d8935d5: Status 404 returned error can't find the container with id 2448ac9953edde68c3d5b1415b0d3af6ab3a755afbb8ae03ab27edc90d8935d5 Nov 02 23:54:09.200218 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:09.200180 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68467d9546-dzddb" event={"ID":"65550239-10be-4c4e-b1fb-b7e7235bdb49","Type":"ContainerStarted","Data":"c1a48d7cffe370f9f094c566069fee8b9dd2b311d0d8ac15105c85d44c774ea8"} Nov 02 23:54:09.200637 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:09.200237 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68467d9546-dzddb" event={"ID":"65550239-10be-4c4e-b1fb-b7e7235bdb49","Type":"ContainerStarted","Data":"2448ac9953edde68c3d5b1415b0d3af6ab3a755afbb8ae03ab27edc90d8935d5"} Nov 02 23:54:09.200637 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:09.200287 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-68467d9546-dzddb" Nov 02 23:54:09.317562 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:09.317513 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-68467d9546-dzddb" podStartSLOduration=1.317499631 podStartE2EDuration="1.317499631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:54:09.315396065 +0000 UTC m=+3935.553893744" watchObservedRunningTime="2026-11-02 23:54:09.317499631 +0000 UTC m=+3935.555997293" Nov 02 23:54:11.737044 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:11.737011 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-68467d9546-dzddb"] Nov 02 23:54:16.737843 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:16.737762 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-68467d9546-dzddb" podUID="65550239-10be-4c4e-b1fb-b7e7235bdb49" containerName="payload-processing" containerID="cri-o://c1a48d7cffe370f9f094c566069fee8b9dd2b311d0d8ac15105c85d44c774ea8" gracePeriod=25 Nov 02 23:54:17.009419 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:17.009362 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68467d9546-dzddb" Nov 02 23:54:17.130995 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:17.130965 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65550239-10be-4c4e-b1fb-b7e7235bdb49-kube-api-access-nvcgc\" (UniqueName: \"kubernetes.io/projected/65550239-10be-4c4e-b1fb-b7e7235bdb49-kube-api-access-nvcgc\") pod \"65550239-10be-4c4e-b1fb-b7e7235bdb49\" (UID: \"65550239-10be-4c4e-b1fb-b7e7235bdb49\") " Nov 02 23:54:17.130995 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:17.131001 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65550239-10be-4c4e-b1fb-b7e7235bdb49-provider-credentials\" (UniqueName: \"kubernetes.io/projected/65550239-10be-4c4e-b1fb-b7e7235bdb49-provider-credentials\") pod \"65550239-10be-4c4e-b1fb-b7e7235bdb49\" (UID: \"65550239-10be-4c4e-b1fb-b7e7235bdb49\") " Nov 02 23:54:17.131248 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:17.131083 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65550239-10be-4c4e-b1fb-b7e7235bdb49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65550239-10be-4c4e-b1fb-b7e7235bdb49-plugins-config-volume\") pod \"65550239-10be-4c4e-b1fb-b7e7235bdb49\" (UID: \"65550239-10be-4c4e-b1fb-b7e7235bdb49\") " Nov 02 23:54:17.131248 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:17.131109 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65550239-10be-4c4e-b1fb-b7e7235bdb49-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/65550239-10be-4c4e-b1fb-b7e7235bdb49-routing-overlay-volume\") pod \"65550239-10be-4c4e-b1fb-b7e7235bdb49\" (UID: \"65550239-10be-4c4e-b1fb-b7e7235bdb49\") " Nov 02 23:54:17.131551 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:17.131520 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65550239-10be-4c4e-b1fb-b7e7235bdb49-plugins-config-volume" pod "65550239-10be-4c4e-b1fb-b7e7235bdb49" (UID: "65550239-10be-4c4e-b1fb-b7e7235bdb49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:54:17.131551 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:17.131524 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65550239-10be-4c4e-b1fb-b7e7235bdb49-routing-overlay-volume" pod "65550239-10be-4c4e-b1fb-b7e7235bdb49" (UID: "65550239-10be-4c4e-b1fb-b7e7235bdb49"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:54:17.131685 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:17.131644 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65550239-10be-4c4e-b1fb-b7e7235bdb49-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:54:17.131685 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:17.131660 5727 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/65550239-10be-4c4e-b1fb-b7e7235bdb49-routing-overlay-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:54:17.133499 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:17.133469 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65550239-10be-4c4e-b1fb-b7e7235bdb49-kube-api-access-nvcgc" pod "65550239-10be-4c4e-b1fb-b7e7235bdb49" (UID: "65550239-10be-4c4e-b1fb-b7e7235bdb49"). InnerVolumeSpecName "kube-api-access-nvcgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:54:17.133783 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:17.133755 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65550239-10be-4c4e-b1fb-b7e7235bdb49-provider-credentials" pod "65550239-10be-4c4e-b1fb-b7e7235bdb49" (UID: "65550239-10be-4c4e-b1fb-b7e7235bdb49"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:54:17.232511 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:17.232479 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvcgc\" (UniqueName: \"kubernetes.io/projected/65550239-10be-4c4e-b1fb-b7e7235bdb49-kube-api-access-nvcgc\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:54:17.232511 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:17.232507 5727 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/65550239-10be-4c4e-b1fb-b7e7235bdb49-provider-credentials\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:54:17.272565 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:17.272489 5727 generic.go:356] "Generic (PLEG): container finished" podID="65550239-10be-4c4e-b1fb-b7e7235bdb49" containerID="c1a48d7cffe370f9f094c566069fee8b9dd2b311d0d8ac15105c85d44c774ea8" exitCode=0 Nov 02 23:54:17.272565 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:17.272541 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68467d9546-dzddb" Nov 02 23:54:17.272770 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:17.272571 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68467d9546-dzddb" event={"ID":"65550239-10be-4c4e-b1fb-b7e7235bdb49","Type":"ContainerDied","Data":"c1a48d7cffe370f9f094c566069fee8b9dd2b311d0d8ac15105c85d44c774ea8"} Nov 02 23:54:17.272770 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:17.272628 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68467d9546-dzddb" event={"ID":"65550239-10be-4c4e-b1fb-b7e7235bdb49","Type":"ContainerDied","Data":"2448ac9953edde68c3d5b1415b0d3af6ab3a755afbb8ae03ab27edc90d8935d5"} Nov 02 23:54:17.272770 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:17.272652 5727 scope.go:122] "RemoveContainer" containerID="c1a48d7cffe370f9f094c566069fee8b9dd2b311d0d8ac15105c85d44c774ea8" Nov 02 23:54:17.287437 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:17.287415 5727 scope.go:122] "RemoveContainer" containerID="c1a48d7cffe370f9f094c566069fee8b9dd2b311d0d8ac15105c85d44c774ea8" Nov 02 23:54:17.287717 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:54:17.287698 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1a48d7cffe370f9f094c566069fee8b9dd2b311d0d8ac15105c85d44c774ea8\": container with ID starting with c1a48d7cffe370f9f094c566069fee8b9dd2b311d0d8ac15105c85d44c774ea8 not found: ID does not exist" containerID="c1a48d7cffe370f9f094c566069fee8b9dd2b311d0d8ac15105c85d44c774ea8" Nov 02 23:54:17.287776 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:17.287726 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1a48d7cffe370f9f094c566069fee8b9dd2b311d0d8ac15105c85d44c774ea8"} err="failed to get container status \"c1a48d7cffe370f9f094c566069fee8b9dd2b311d0d8ac15105c85d44c774ea8\": rpc error: code = NotFound desc = could not find container \"c1a48d7cffe370f9f094c566069fee8b9dd2b311d0d8ac15105c85d44c774ea8\": container with ID starting with c1a48d7cffe370f9f094c566069fee8b9dd2b311d0d8ac15105c85d44c774ea8 not found: ID does not exist" Nov 02 23:54:17.437731 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:17.437699 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-68467d9546-dzddb"] Nov 02 23:54:17.443299 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:17.443265 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-68467d9546-dzddb"] Nov 02 23:54:17.883508 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:17.883477 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65550239-10be-4c4e-b1fb-b7e7235bdb49" path="/var/lib/kubelet/pods/65550239-10be-4c4e-b1fb-b7e7235bdb49/volumes" Nov 02 23:54:20.652586 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:20.652546 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-78b869747-rpl9n"] Nov 02 23:54:20.653426 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:20.653404 5727 cpu_manager.go:414] "containerMap: removing container" podUID="65550239-10be-4c4e-b1fb-b7e7235bdb49" containerName="payload-processing" Nov 02 23:54:20.653548 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:20.653426 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65550239-10be-4c4e-b1fb-b7e7235bdb49" containerName="payload-processing" Nov 02 23:54:20.653608 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:20.653600 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="65550239-10be-4c4e-b1fb-b7e7235bdb49" containerName="payload-processing" Nov 02 23:54:20.658547 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:20.658524 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78b869747-rpl9n" Nov 02 23:54:20.718475 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:20.718446 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 23:54:20.718661 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:20.718447 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 23:54:20.767664 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:20.767626 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 23:54:20.767664 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:20.767624 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 23:54:20.767909 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:20.767833 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4f32f503d9a7\"" Nov 02 23:54:20.768087 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:20.768067 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-7wxb5\"" Nov 02 23:54:20.771638 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:20.771616 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-78b869747-rpl9n"] Nov 02 23:54:20.771787 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:20.771690 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr659\" (UniqueName: \"kubernetes.io/projected/f24353b6-68bc-49e7-abf1-2ee1d0bb2074-kube-api-access-rr659\") pod \"payload-processing-external-model-78b869747-rpl9n\" (UID: \"f24353b6-68bc-49e7-abf1-2ee1d0bb2074\") " pod="models-as-a-service/payload-processing-external-model-78b869747-rpl9n" Nov 02 23:54:20.771858 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:20.771801 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f24353b6-68bc-49e7-abf1-2ee1d0bb2074-routing-overlay-volume\") pod \"payload-processing-external-model-78b869747-rpl9n\" (UID: \"f24353b6-68bc-49e7-abf1-2ee1d0bb2074\") " pod="models-as-a-service/payload-processing-external-model-78b869747-rpl9n" Nov 02 23:54:20.771917 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:20.771877 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f24353b6-68bc-49e7-abf1-2ee1d0bb2074-plugins-config-volume\") pod \"payload-processing-external-model-78b869747-rpl9n\" (UID: \"f24353b6-68bc-49e7-abf1-2ee1d0bb2074\") " pod="models-as-a-service/payload-processing-external-model-78b869747-rpl9n" Nov 02 23:54:20.771917 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:20.771893 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f24353b6-68bc-49e7-abf1-2ee1d0bb2074-provider-credentials\") pod \"payload-processing-external-model-78b869747-rpl9n\" (UID: \"f24353b6-68bc-49e7-abf1-2ee1d0bb2074\") " pod="models-as-a-service/payload-processing-external-model-78b869747-rpl9n" Nov 02 23:54:20.872755 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:20.872715 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f24353b6-68bc-49e7-abf1-2ee1d0bb2074-plugins-config-volume\") pod \"payload-processing-external-model-78b869747-rpl9n\" (UID: \"f24353b6-68bc-49e7-abf1-2ee1d0bb2074\") " pod="models-as-a-service/payload-processing-external-model-78b869747-rpl9n" Nov 02 23:54:20.872755 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:20.872750 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f24353b6-68bc-49e7-abf1-2ee1d0bb2074-provider-credentials\") pod \"payload-processing-external-model-78b869747-rpl9n\" (UID: \"f24353b6-68bc-49e7-abf1-2ee1d0bb2074\") " pod="models-as-a-service/payload-processing-external-model-78b869747-rpl9n" Nov 02 23:54:20.873008 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:20.872820 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr659\" (UniqueName: \"kubernetes.io/projected/f24353b6-68bc-49e7-abf1-2ee1d0bb2074-kube-api-access-rr659\") pod \"payload-processing-external-model-78b869747-rpl9n\" (UID: \"f24353b6-68bc-49e7-abf1-2ee1d0bb2074\") " pod="models-as-a-service/payload-processing-external-model-78b869747-rpl9n" Nov 02 23:54:20.873008 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:20.872885 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f24353b6-68bc-49e7-abf1-2ee1d0bb2074-routing-overlay-volume\") pod \"payload-processing-external-model-78b869747-rpl9n\" (UID: \"f24353b6-68bc-49e7-abf1-2ee1d0bb2074\") " pod="models-as-a-service/payload-processing-external-model-78b869747-rpl9n" Nov 02 23:54:20.873008 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:54:20.872991 5727 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-4f32f503d9a7: secret "agc-cred-4f32f503d9a7" not found Nov 02 23:54:20.873008 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:54:20.873009 5727 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-78b869747-rpl9n: secret "agc-cred-4f32f503d9a7" not found Nov 02 23:54:20.873254 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:54:20.873080 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f24353b6-68bc-49e7-abf1-2ee1d0bb2074-provider-credentials podName:f24353b6-68bc-49e7-abf1-2ee1d0bb2074 nodeName:}" failed. No retries permitted until 2026-11-02 23:54:21.373060846 +0000 UTC m=+3947.611558503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f24353b6-68bc-49e7-abf1-2ee1d0bb2074-provider-credentials") pod "payload-processing-external-model-78b869747-rpl9n" (UID: "f24353b6-68bc-49e7-abf1-2ee1d0bb2074") : secret "agc-cred-4f32f503d9a7" not found Nov 02 23:54:20.873579 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:20.873555 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f24353b6-68bc-49e7-abf1-2ee1d0bb2074-routing-overlay-volume\") pod \"payload-processing-external-model-78b869747-rpl9n\" (UID: \"f24353b6-68bc-49e7-abf1-2ee1d0bb2074\") " pod="models-as-a-service/payload-processing-external-model-78b869747-rpl9n" Nov 02 23:54:20.873736 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:20.873712 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f24353b6-68bc-49e7-abf1-2ee1d0bb2074-plugins-config-volume\") pod \"payload-processing-external-model-78b869747-rpl9n\" (UID: \"f24353b6-68bc-49e7-abf1-2ee1d0bb2074\") " pod="models-as-a-service/payload-processing-external-model-78b869747-rpl9n" Nov 02 23:54:20.934025 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:20.933957 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr659\" (UniqueName: \"kubernetes.io/projected/f24353b6-68bc-49e7-abf1-2ee1d0bb2074-kube-api-access-rr659\") pod \"payload-processing-external-model-78b869747-rpl9n\" (UID: \"f24353b6-68bc-49e7-abf1-2ee1d0bb2074\") " pod="models-as-a-service/payload-processing-external-model-78b869747-rpl9n" Nov 02 23:54:21.379583 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:21.379546 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f24353b6-68bc-49e7-abf1-2ee1d0bb2074-provider-credentials\") pod \"payload-processing-external-model-78b869747-rpl9n\" (UID: \"f24353b6-68bc-49e7-abf1-2ee1d0bb2074\") " pod="models-as-a-service/payload-processing-external-model-78b869747-rpl9n" Nov 02 23:54:21.379793 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:54:21.379712 5727 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-4f32f503d9a7: secret "agc-cred-4f32f503d9a7" not found Nov 02 23:54:21.379793 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:54:21.379734 5727 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-78b869747-rpl9n: secret "agc-cred-4f32f503d9a7" not found Nov 02 23:54:21.379927 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:54:21.379799 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f24353b6-68bc-49e7-abf1-2ee1d0bb2074-provider-credentials podName:f24353b6-68bc-49e7-abf1-2ee1d0bb2074 nodeName:}" failed. No retries permitted until 2026-11-02 23:54:22.379777583 +0000 UTC m=+3948.618275239 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f24353b6-68bc-49e7-abf1-2ee1d0bb2074-provider-credentials") pod "payload-processing-external-model-78b869747-rpl9n" (UID: "f24353b6-68bc-49e7-abf1-2ee1d0bb2074") : secret "agc-cred-4f32f503d9a7" not found Nov 02 23:54:21.544259 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:21.544222 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-78b869747-rpl9n"] Nov 02 23:54:21.544606 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:54:21.544585 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="models-as-a-service/payload-processing-external-model-78b869747-rpl9n" podUID="f24353b6-68bc-49e7-abf1-2ee1d0bb2074" Nov 02 23:54:22.309357 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:22.309327 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78b869747-rpl9n" Nov 02 23:54:22.318191 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:22.318162 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78b869747-rpl9n" Nov 02 23:54:22.395169 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:22.395110 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f24353b6-68bc-49e7-abf1-2ee1d0bb2074-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f24353b6-68bc-49e7-abf1-2ee1d0bb2074-plugins-config-volume\") pod \"f24353b6-68bc-49e7-abf1-2ee1d0bb2074\" (UID: \"f24353b6-68bc-49e7-abf1-2ee1d0bb2074\") " Nov 02 23:54:22.396000 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:22.395974 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f24353b6-68bc-49e7-abf1-2ee1d0bb2074-kube-api-access-rr659\" (UniqueName: \"kubernetes.io/projected/f24353b6-68bc-49e7-abf1-2ee1d0bb2074-kube-api-access-rr659\") pod \"f24353b6-68bc-49e7-abf1-2ee1d0bb2074\" (UID: \"f24353b6-68bc-49e7-abf1-2ee1d0bb2074\") " Nov 02 23:54:22.396644 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:22.396624 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f24353b6-68bc-49e7-abf1-2ee1d0bb2074-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f24353b6-68bc-49e7-abf1-2ee1d0bb2074-routing-overlay-volume\") pod \"f24353b6-68bc-49e7-abf1-2ee1d0bb2074\" (UID: \"f24353b6-68bc-49e7-abf1-2ee1d0bb2074\") " Nov 02 23:54:22.397735 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:22.397708 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f24353b6-68bc-49e7-abf1-2ee1d0bb2074-provider-credentials\") pod \"payload-processing-external-model-78b869747-rpl9n\" (UID: \"f24353b6-68bc-49e7-abf1-2ee1d0bb2074\") " pod="models-as-a-service/payload-processing-external-model-78b869747-rpl9n" Nov 02 23:54:22.398428 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:22.395832 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f24353b6-68bc-49e7-abf1-2ee1d0bb2074-plugins-config-volume" pod "f24353b6-68bc-49e7-abf1-2ee1d0bb2074" (UID: "f24353b6-68bc-49e7-abf1-2ee1d0bb2074"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:54:22.398553 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:22.397148 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f24353b6-68bc-49e7-abf1-2ee1d0bb2074-routing-overlay-volume" pod "f24353b6-68bc-49e7-abf1-2ee1d0bb2074" (UID: "f24353b6-68bc-49e7-abf1-2ee1d0bb2074"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:54:22.398664 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:54:22.397998 5727 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-4f32f503d9a7: secret "agc-cred-4f32f503d9a7" not found Nov 02 23:54:22.398759 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:54:22.398747 5727 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-78b869747-rpl9n: secret "agc-cred-4f32f503d9a7" not found Nov 02 23:54:22.398904 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:54:22.398893 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f24353b6-68bc-49e7-abf1-2ee1d0bb2074-provider-credentials podName:f24353b6-68bc-49e7-abf1-2ee1d0bb2074 nodeName:}" failed. No retries permitted until 2026-11-02 23:54:24.398868655 +0000 UTC m=+3950.637366308 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f24353b6-68bc-49e7-abf1-2ee1d0bb2074-provider-credentials") pod "payload-processing-external-model-78b869747-rpl9n" (UID: "f24353b6-68bc-49e7-abf1-2ee1d0bb2074") : secret "agc-cred-4f32f503d9a7" not found Nov 02 23:54:22.405645 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:22.405622 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f24353b6-68bc-49e7-abf1-2ee1d0bb2074-kube-api-access-rr659" pod "f24353b6-68bc-49e7-abf1-2ee1d0bb2074" (UID: "f24353b6-68bc-49e7-abf1-2ee1d0bb2074"). InnerVolumeSpecName "kube-api-access-rr659". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:54:22.499244 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:22.499209 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr659\" (UniqueName: \"kubernetes.io/projected/f24353b6-68bc-49e7-abf1-2ee1d0bb2074-kube-api-access-rr659\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:54:22.499244 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:22.499240 5727 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f24353b6-68bc-49e7-abf1-2ee1d0bb2074-routing-overlay-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:54:22.499244 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:22.499250 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f24353b6-68bc-49e7-abf1-2ee1d0bb2074-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:54:23.316725 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:23.316689 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78b869747-rpl9n" Nov 02 23:54:23.542586 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:23.542554 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-78b869747-rpl9n"] Nov 02 23:54:23.551023 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:23.550995 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-78b869747-rpl9n"] Nov 02 23:54:23.713257 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:23.713090 5727 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f24353b6-68bc-49e7-abf1-2ee1d0bb2074-provider-credentials\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:54:23.885462 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:23.885430 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f24353b6-68bc-49e7-abf1-2ee1d0bb2074" path="/var/lib/kubelet/pods/f24353b6-68bc-49e7-abf1-2ee1d0bb2074/volumes" Nov 02 23:54:27.847208 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:54:27.847172 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:54:31.865989 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:31.865939 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-18a6d8-55c5858669-hts8r"] Nov 02 23:54:31.911462 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:31.903234 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-18a6d8-55c5858669-hts8r" Nov 02 23:54:31.918679 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:31.918531 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78557250-4ce3-404d-b057-e66bde4fec4d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-18a6d8-55c5858669-hts8r\" (UID: \"78557250-4ce3-404d-b057-e66bde4fec4d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-18a6d8-55c5858669-hts8r" Nov 02 23:54:31.918679 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:31.918605 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fkdc\" (UniqueName: \"kubernetes.io/projected/78557250-4ce3-404d-b057-e66bde4fec4d-kube-api-access-7fkdc\") pod \"payload-processing-e2e-inf-a-18a6d8-55c5858669-hts8r\" (UID: \"78557250-4ce3-404d-b057-e66bde4fec4d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-18a6d8-55c5858669-hts8r" Nov 02 23:54:31.930839 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:31.930812 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-18a6d8-55c5858669-hts8r"] Nov 02 23:54:31.968437 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:31.968401 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-18a6d8-dockercfg-sfct5\"" Nov 02 23:54:31.981431 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:31.981401 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-18a6d8\"" Nov 02 23:54:32.019875 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:32.019834 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fkdc\" (UniqueName: \"kubernetes.io/projected/78557250-4ce3-404d-b057-e66bde4fec4d-kube-api-access-7fkdc\") pod \"payload-processing-e2e-inf-a-18a6d8-55c5858669-hts8r\" (UID: \"78557250-4ce3-404d-b057-e66bde4fec4d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-18a6d8-55c5858669-hts8r" Nov 02 23:54:32.020265 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:32.020239 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78557250-4ce3-404d-b057-e66bde4fec4d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-18a6d8-55c5858669-hts8r\" (UID: \"78557250-4ce3-404d-b057-e66bde4fec4d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-18a6d8-55c5858669-hts8r" Nov 02 23:54:32.021332 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:32.021311 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78557250-4ce3-404d-b057-e66bde4fec4d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-18a6d8-55c5858669-hts8r\" (UID: \"78557250-4ce3-404d-b057-e66bde4fec4d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-18a6d8-55c5858669-hts8r" Nov 02 23:54:32.031297 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:32.031266 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-18a6d8-667d88ff5-fg8ng"] Nov 02 23:54:32.036606 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:32.036556 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18a6d8-667d88ff5-fg8ng" Nov 02 23:54:32.080983 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:32.080951 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fkdc\" (UniqueName: \"kubernetes.io/projected/78557250-4ce3-404d-b057-e66bde4fec4d-kube-api-access-7fkdc\") pod \"payload-processing-e2e-inf-a-18a6d8-55c5858669-hts8r\" (UID: \"78557250-4ce3-404d-b057-e66bde4fec4d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-18a6d8-55c5858669-hts8r" Nov 02 23:54:32.103707 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:32.098880 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-18a6d8-667d88ff5-fg8ng"] Nov 02 23:54:32.122597 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:32.122078 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4ft2\" (UniqueName: \"kubernetes.io/projected/aa393900-e35d-4751-817c-c6ed6a2d6a8a-kube-api-access-s4ft2\") pod \"payload-pre-processing-e2e-inf-a-18a6d8-667d88ff5-fg8ng\" (UID: \"aa393900-e35d-4751-817c-c6ed6a2d6a8a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18a6d8-667d88ff5-fg8ng" Nov 02 23:54:32.122597 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:32.122213 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa393900-e35d-4751-817c-c6ed6a2d6a8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-18a6d8-667d88ff5-fg8ng\" (UID: \"aa393900-e35d-4751-817c-c6ed6a2d6a8a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18a6d8-667d88ff5-fg8ng" Nov 02 23:54:32.223930 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:32.223886 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa393900-e35d-4751-817c-c6ed6a2d6a8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-18a6d8-667d88ff5-fg8ng\" (UID: \"aa393900-e35d-4751-817c-c6ed6a2d6a8a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18a6d8-667d88ff5-fg8ng" Nov 02 23:54:32.224138 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:32.224012 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4ft2\" (UniqueName: \"kubernetes.io/projected/aa393900-e35d-4751-817c-c6ed6a2d6a8a-kube-api-access-s4ft2\") pod \"payload-pre-processing-e2e-inf-a-18a6d8-667d88ff5-fg8ng\" (UID: \"aa393900-e35d-4751-817c-c6ed6a2d6a8a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18a6d8-667d88ff5-fg8ng" Nov 02 23:54:32.224646 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:32.224622 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa393900-e35d-4751-817c-c6ed6a2d6a8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-18a6d8-667d88ff5-fg8ng\" (UID: \"aa393900-e35d-4751-817c-c6ed6a2d6a8a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18a6d8-667d88ff5-fg8ng" Nov 02 23:54:32.233697 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:32.233674 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-18a6d8-55c5858669-hts8r" Nov 02 23:54:32.286175 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:32.285462 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4ft2\" (UniqueName: \"kubernetes.io/projected/aa393900-e35d-4751-817c-c6ed6a2d6a8a-kube-api-access-s4ft2\") pod \"payload-pre-processing-e2e-inf-a-18a6d8-667d88ff5-fg8ng\" (UID: \"aa393900-e35d-4751-817c-c6ed6a2d6a8a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18a6d8-667d88ff5-fg8ng" Nov 02 23:54:32.353415 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:32.353378 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18a6d8-667d88ff5-fg8ng" Nov 02 23:54:32.637724 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:32.637692 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-18a6d8-55c5858669-hts8r"] Nov 02 23:54:32.639301 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:54:32.639273 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78557250_4ce3_404d_b057_e66bde4fec4d.slice/crio-edf0f176d184dc90c2bc2f0e42d10436115921c9812f07e267b3740961d78006 WatchSource:0}: Error finding container edf0f176d184dc90c2bc2f0e42d10436115921c9812f07e267b3740961d78006: Status 404 returned error can't find the container with id edf0f176d184dc90c2bc2f0e42d10436115921c9812f07e267b3740961d78006 Nov 02 23:54:32.786722 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:32.786691 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-18a6d8-667d88ff5-fg8ng"] Nov 02 23:54:32.788045 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:54:32.788006 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa393900_e35d_4751_817c_c6ed6a2d6a8a.slice/crio-7204dffa0996ce315ac846012b22c42b26711fd9bab5060da8cdea363cbe0a4f WatchSource:0}: Error finding container 7204dffa0996ce315ac846012b22c42b26711fd9bab5060da8cdea363cbe0a4f: Status 404 returned error can't find the container with id 7204dffa0996ce315ac846012b22c42b26711fd9bab5060da8cdea363cbe0a4f Nov 02 23:54:33.413446 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:33.413411 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-18a6d8-55c5858669-hts8r" event={"ID":"78557250-4ce3-404d-b057-e66bde4fec4d","Type":"ContainerStarted","Data":"67d22fd482365be5fd05bbb22c120ed0775825ac7e6f23019f5ad79bdecfcb80"} Nov 02 23:54:33.413917 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:33.413464 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-18a6d8-55c5858669-hts8r" event={"ID":"78557250-4ce3-404d-b057-e66bde4fec4d","Type":"ContainerStarted","Data":"edf0f176d184dc90c2bc2f0e42d10436115921c9812f07e267b3740961d78006"} Nov 02 23:54:33.413917 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:33.413502 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-18a6d8-55c5858669-hts8r" Nov 02 23:54:33.415374 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:33.415344 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18a6d8-667d88ff5-fg8ng" event={"ID":"aa393900-e35d-4751-817c-c6ed6a2d6a8a","Type":"ContainerStarted","Data":"7a920323cd722507ecbaf8991f8e3a3eda8bcb225976560c6e3185e134293f6d"} Nov 02 23:54:33.415513 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:33.415394 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18a6d8-667d88ff5-fg8ng" event={"ID":"aa393900-e35d-4751-817c-c6ed6a2d6a8a","Type":"ContainerStarted","Data":"7204dffa0996ce315ac846012b22c42b26711fd9bab5060da8cdea363cbe0a4f"} Nov 02 23:54:33.415513 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:33.415439 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18a6d8-667d88ff5-fg8ng" Nov 02 23:54:33.552917 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:33.552865 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-18a6d8-55c5858669-hts8r" podStartSLOduration=2.552853201 podStartE2EDuration="2.552853201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:54:33.530590979 +0000 UTC m=+3959.769088642" watchObservedRunningTime="2026-11-02 23:54:33.552853201 +0000 UTC m=+3959.791350864" Nov 02 23:54:33.688346 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:33.688214 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18a6d8-667d88ff5-fg8ng" podStartSLOduration=2.688193197 podStartE2EDuration="2.688193197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:54:33.686960509 +0000 UTC m=+3959.925458196" watchObservedRunningTime="2026-11-02 23:54:33.688193197 +0000 UTC m=+3959.926690878" Nov 02 23:54:35.912164 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:35.912107 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-942137-7b896967c-j7p9z"] Nov 02 23:54:35.912586 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:35.912334 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-942137-7b896967c-j7p9z" podUID="163e0b93-962f-4506-9465-c6ed52fe99b4" containerName="maas-api" containerID="cri-o://a7a89e6d9a0606e09fa0c64f58d742bfdcfc5248d20041dc450068513242be75" gracePeriod=30 Nov 02 23:54:36.230578 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:36.230550 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-942137-7b896967c-j7p9z" Nov 02 23:54:36.294151 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:36.293536 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/163e0b93-962f-4506-9465-c6ed52fe99b4-kube-api-access-qkkc4\" (UniqueName: \"kubernetes.io/projected/163e0b93-962f-4506-9465-c6ed52fe99b4-kube-api-access-qkkc4\") pod \"163e0b93-962f-4506-9465-c6ed52fe99b4\" (UID: \"163e0b93-962f-4506-9465-c6ed52fe99b4\") " Nov 02 23:54:36.294151 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:36.293614 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/163e0b93-962f-4506-9465-c6ed52fe99b4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/163e0b93-962f-4506-9465-c6ed52fe99b4-metrics-certs\") pod \"163e0b93-962f-4506-9465-c6ed52fe99b4\" (UID: \"163e0b93-962f-4506-9465-c6ed52fe99b4\") " Nov 02 23:54:36.294151 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:36.293684 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/163e0b93-962f-4506-9465-c6ed52fe99b4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/163e0b93-962f-4506-9465-c6ed52fe99b4-maas-api-tls\") pod \"163e0b93-962f-4506-9465-c6ed52fe99b4\" (UID: \"163e0b93-962f-4506-9465-c6ed52fe99b4\") " Nov 02 23:54:36.294151 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:36.293775 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/163e0b93-962f-4506-9465-c6ed52fe99b4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/163e0b93-962f-4506-9465-c6ed52fe99b4-openshift-service-ca\") pod \"163e0b93-962f-4506-9465-c6ed52fe99b4\" (UID: \"163e0b93-962f-4506-9465-c6ed52fe99b4\") " Nov 02 23:54:36.309985 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:36.294740 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/163e0b93-962f-4506-9465-c6ed52fe99b4-openshift-service-ca" pod "163e0b93-962f-4506-9465-c6ed52fe99b4" (UID: "163e0b93-962f-4506-9465-c6ed52fe99b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:54:36.323923 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:36.323867 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/163e0b93-962f-4506-9465-c6ed52fe99b4-metrics-certs" pod "163e0b93-962f-4506-9465-c6ed52fe99b4" (UID: "163e0b93-962f-4506-9465-c6ed52fe99b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:54:36.324223 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:36.324186 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/163e0b93-962f-4506-9465-c6ed52fe99b4-kube-api-access-qkkc4" pod "163e0b93-962f-4506-9465-c6ed52fe99b4" (UID: "163e0b93-962f-4506-9465-c6ed52fe99b4"). InnerVolumeSpecName "kube-api-access-qkkc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:54:36.342185 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:36.340717 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/163e0b93-962f-4506-9465-c6ed52fe99b4-maas-api-tls" pod "163e0b93-962f-4506-9465-c6ed52fe99b4" (UID: "163e0b93-962f-4506-9465-c6ed52fe99b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:54:36.432431 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:36.427958 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkkc4\" (UniqueName: \"kubernetes.io/projected/163e0b93-962f-4506-9465-c6ed52fe99b4-kube-api-access-qkkc4\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:54:36.432431 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:36.427994 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/163e0b93-962f-4506-9465-c6ed52fe99b4-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:54:36.432431 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:36.428008 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/163e0b93-962f-4506-9465-c6ed52fe99b4-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:54:36.432431 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:36.428022 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/163e0b93-962f-4506-9465-c6ed52fe99b4-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:54:36.523217 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:36.510317 5727 generic.go:356] "Generic (PLEG): container finished" podID="163e0b93-962f-4506-9465-c6ed52fe99b4" containerID="a7a89e6d9a0606e09fa0c64f58d742bfdcfc5248d20041dc450068513242be75" exitCode=0 Nov 02 23:54:36.523217 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:36.510595 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-942137-7b896967c-j7p9z" event={"ID":"163e0b93-962f-4506-9465-c6ed52fe99b4","Type":"ContainerDied","Data":"a7a89e6d9a0606e09fa0c64f58d742bfdcfc5248d20041dc450068513242be75"} Nov 02 23:54:36.523217 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:36.510652 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-942137-7b896967c-j7p9z" event={"ID":"163e0b93-962f-4506-9465-c6ed52fe99b4","Type":"ContainerDied","Data":"b15741ccb288f4c654b7b5c0ff16d03a73394875fa22eb7e487982b4287c043b"} Nov 02 23:54:36.523217 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:36.510703 5727 scope.go:122] "RemoveContainer" containerID="a7a89e6d9a0606e09fa0c64f58d742bfdcfc5248d20041dc450068513242be75" Nov 02 23:54:36.523217 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:36.510914 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-942137-7b896967c-j7p9z" Nov 02 23:54:36.644386 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:36.644267 5727 scope.go:122] "RemoveContainer" containerID="a7a89e6d9a0606e09fa0c64f58d742bfdcfc5248d20041dc450068513242be75" Nov 02 23:54:36.652518 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:54:36.652424 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7a89e6d9a0606e09fa0c64f58d742bfdcfc5248d20041dc450068513242be75\": container with ID starting with a7a89e6d9a0606e09fa0c64f58d742bfdcfc5248d20041dc450068513242be75 not found: ID does not exist" containerID="a7a89e6d9a0606e09fa0c64f58d742bfdcfc5248d20041dc450068513242be75" Nov 02 23:54:36.652518 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:36.652484 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7a89e6d9a0606e09fa0c64f58d742bfdcfc5248d20041dc450068513242be75"} err="failed to get container status \"a7a89e6d9a0606e09fa0c64f58d742bfdcfc5248d20041dc450068513242be75\": rpc error: code = NotFound desc = could not find container \"a7a89e6d9a0606e09fa0c64f58d742bfdcfc5248d20041dc450068513242be75\": container with ID starting with a7a89e6d9a0606e09fa0c64f58d742bfdcfc5248d20041dc450068513242be75 not found: ID does not exist" Nov 02 23:54:36.762856 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:36.762825 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-942137-7b896967c-j7p9z"] Nov 02 23:54:36.767440 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:36.767405 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-942137-7b896967c-j7p9z"] Nov 02 23:54:37.884416 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:37.884384 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="163e0b93-962f-4506-9465-c6ed52fe99b4" path="/var/lib/kubelet/pods/163e0b93-962f-4506-9465-c6ed52fe99b4/volumes" Nov 02 23:54:44.154438 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.154361 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d"] Nov 02 23:54:44.155039 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.155023 5727 cpu_manager.go:414] "containerMap: removing container" podUID="163e0b93-962f-4506-9465-c6ed52fe99b4" containerName="maas-api" Nov 02 23:54:44.155039 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.155039 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="163e0b93-962f-4506-9465-c6ed52fe99b4" containerName="maas-api" Nov 02 23:54:44.155212 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.155199 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="163e0b93-962f-4506-9465-c6ed52fe99b4" containerName="maas-api" Nov 02 23:54:44.158906 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.158888 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:44.240780 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.240734 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94871baf-708f-4190-b685-87f0e8ece5d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-znv9d\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:44.240961 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.240821 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-znv9d\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:44.240961 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.240896 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-znv9d\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:44.241076 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.240951 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94871baf-708f-4190-b685-87f0e8ece5d4-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-znv9d\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:44.241076 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.241010 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-znv9d\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:44.241076 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.241036 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94871baf-708f-4190-b685-87f0e8ece5d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-znv9d\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:44.241249 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.241076 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-znv9d\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:44.241249 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.241194 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-znv9d\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:44.241335 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.241276 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj4x2\" (UniqueName: \"kubernetes.io/projected/94871baf-708f-4190-b685-87f0e8ece5d4-kube-api-access-tj4x2\") pod \"maas-default-gateway-openshift-default-694db55c87-znv9d\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:44.306882 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.306848 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d"] Nov 02 23:54:44.342498 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.342467 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj4x2\" (UniqueName: \"kubernetes.io/projected/94871baf-708f-4190-b685-87f0e8ece5d4-kube-api-access-tj4x2\") pod \"maas-default-gateway-openshift-default-694db55c87-znv9d\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:44.342680 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.342531 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94871baf-708f-4190-b685-87f0e8ece5d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-znv9d\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:44.342680 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.342568 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-znv9d\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:44.342680 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.342604 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-znv9d\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:44.342680 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.342626 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94871baf-708f-4190-b685-87f0e8ece5d4-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-znv9d\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:44.342910 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.342687 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-znv9d\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:44.342910 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.342721 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94871baf-708f-4190-b685-87f0e8ece5d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-znv9d\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:44.342910 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.342783 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-znv9d\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:44.342910 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.342814 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-znv9d\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:44.343148 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.343023 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-znv9d\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:44.343219 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.343183 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-znv9d\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:44.343279 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.343245 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-znv9d\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:44.343370 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.343345 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-znv9d\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:44.343475 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.343455 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94871baf-708f-4190-b685-87f0e8ece5d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-znv9d\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:44.345275 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.345252 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-znv9d\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:44.345470 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.345452 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94871baf-708f-4190-b685-87f0e8ece5d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-znv9d\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:44.403321 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.403292 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj4x2\" (UniqueName: \"kubernetes.io/projected/94871baf-708f-4190-b685-87f0e8ece5d4-kube-api-access-tj4x2\") pod \"maas-default-gateway-openshift-default-694db55c87-znv9d\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:44.419234 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.419179 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94871baf-708f-4190-b685-87f0e8ece5d4-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-znv9d\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:44.425984 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.425961 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-18a6d8-55c5858669-hts8r" Nov 02 23:54:44.426251 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.426233 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18a6d8-667d88ff5-fg8ng" Nov 02 23:54:44.474089 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.474057 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:44.909384 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.909354 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d"] Nov 02 23:54:44.910506 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:54:44.910479 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94871baf_708f_4190_b685_87f0e8ece5d4.slice/crio-f1de949bea52ba9b856b259f42dff08aab101cc9d0b15e5559613c5512a0183b WatchSource:0}: Error finding container f1de949bea52ba9b856b259f42dff08aab101cc9d0b15e5559613c5512a0183b: Status 404 returned error can't find the container with id f1de949bea52ba9b856b259f42dff08aab101cc9d0b15e5559613c5512a0183b Nov 02 23:54:44.913276 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.913248 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:54:44.913348 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.913312 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:54:44.913348 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:44.913342 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:54:45.608538 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:45.608494 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" event={"ID":"94871baf-708f-4190-b685-87f0e8ece5d4","Type":"ContainerStarted","Data":"2f7dfe058b645fe827e29244e08cdffe022084d2ba00a74e0972f8a8886c6850"} Nov 02 23:54:45.608975 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:45.608609 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" event={"ID":"94871baf-708f-4190-b685-87f0e8ece5d4","Type":"ContainerStarted","Data":"f1de949bea52ba9b856b259f42dff08aab101cc9d0b15e5559613c5512a0183b"} Nov 02 23:54:45.736269 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:45.736212 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" podStartSLOduration=1.736198975 podStartE2EDuration="1.736198975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:54:45.73312205 +0000 UTC m=+3971.971619733" watchObservedRunningTime="2026-11-02 23:54:45.736198975 +0000 UTC m=+3971.974696637" Nov 02 23:54:46.478017 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:46.476523 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:46.480277 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:46.480245 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-znv9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 02 23:54:46.480501 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:46.480473 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" podUID="94871baf-708f-4190-b685-87f0e8ece5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 02 23:54:47.479525 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:47.479486 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-znv9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 02 23:54:47.480151 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:47.480093 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" podUID="94871baf-708f-4190-b685-87f0e8ece5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 02 23:54:48.476524 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:48.476486 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-znv9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 02 23:54:48.476810 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:48.476783 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" podUID="94871baf-708f-4190-b685-87f0e8ece5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 02 23:54:49.475529 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:49.475489 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-znv9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 02 23:54:49.476055 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:49.475551 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" podUID="94871baf-708f-4190-b685-87f0e8ece5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 02 23:54:50.475522 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:50.475468 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-znv9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 02 23:54:50.475708 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:50.475535 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" podUID="94871baf-708f-4190-b685-87f0e8ece5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 02 23:54:51.474785 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:51.474747 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-znv9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 02 23:54:51.474959 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:51.474826 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" podUID="94871baf-708f-4190-b685-87f0e8ece5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 02 23:54:52.474633 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:52.474599 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-znv9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 02 23:54:52.474992 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:52.474657 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" podUID="94871baf-708f-4190-b685-87f0e8ece5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 02 23:54:53.516621 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:53.516580 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:53.517636 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:53.517615 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:53.518486 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:54:53.518463 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 02 23:54:57.846400 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:54:57.846363 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:55:14.067804 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:14.067768 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5e7e9e-66b9694f59-5rtsh"] Nov 02 23:55:14.073281 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:14.073259 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5e7e9e-66b9694f59-5rtsh" Nov 02 23:55:14.128008 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:14.127980 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5e7e9e-66b9694f59-5rtsh"] Nov 02 23:55:14.173486 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:14.173457 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-5e7e9e-dockercfg-ngbm8\"" Nov 02 23:55:14.173661 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:14.173504 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-5e7e9e\"" Nov 02 23:55:14.183759 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:14.183723 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2tqh\" (UniqueName: \"kubernetes.io/projected/432f3e59-fed9-4cc8-9e48-403f02f0c40e-kube-api-access-f2tqh\") pod \"payload-pre-processing-e2e-extmod-b-5e7e9e-66b9694f59-5rtsh\" (UID: \"432f3e59-fed9-4cc8-9e48-403f02f0c40e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5e7e9e-66b9694f59-5rtsh" Nov 02 23:55:14.183909 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:14.183784 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432f3e59-fed9-4cc8-9e48-403f02f0c40e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-5e7e9e-66b9694f59-5rtsh\" (UID: \"432f3e59-fed9-4cc8-9e48-403f02f0c40e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5e7e9e-66b9694f59-5rtsh" Nov 02 23:55:14.217777 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:14.217744 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5e7e9e-6cc6b59469-q88rm"] Nov 02 23:55:14.222243 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:14.222221 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-5e7e9e-6cc6b59469-q88rm" Nov 02 23:55:14.280468 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:14.280431 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5e7e9e-6cc6b59469-q88rm"] Nov 02 23:55:14.285818 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:14.285785 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432f3e59-fed9-4cc8-9e48-403f02f0c40e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-5e7e9e-66b9694f59-5rtsh\" (UID: \"432f3e59-fed9-4cc8-9e48-403f02f0c40e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5e7e9e-66b9694f59-5rtsh" Nov 02 23:55:14.286171 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:14.286152 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wmch\" (UniqueName: \"kubernetes.io/projected/05c07885-2e05-4f96-b72d-2e2a3d918bc5-kube-api-access-7wmch\") pod \"payload-processing-e2e-extmod-b-5e7e9e-6cc6b59469-q88rm\" (UID: \"05c07885-2e05-4f96-b72d-2e2a3d918bc5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5e7e9e-6cc6b59469-q88rm" Nov 02 23:55:14.286414 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:14.286395 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c07885-2e05-4f96-b72d-2e2a3d918bc5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-5e7e9e-6cc6b59469-q88rm\" (UID: \"05c07885-2e05-4f96-b72d-2e2a3d918bc5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5e7e9e-6cc6b59469-q88rm" Nov 02 23:55:14.286579 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:14.286563 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2tqh\" (UniqueName: \"kubernetes.io/projected/432f3e59-fed9-4cc8-9e48-403f02f0c40e-kube-api-access-f2tqh\") pod \"payload-pre-processing-e2e-extmod-b-5e7e9e-66b9694f59-5rtsh\" (UID: \"432f3e59-fed9-4cc8-9e48-403f02f0c40e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5e7e9e-66b9694f59-5rtsh" Nov 02 23:55:14.288313 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:14.288264 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432f3e59-fed9-4cc8-9e48-403f02f0c40e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-5e7e9e-66b9694f59-5rtsh\" (UID: \"432f3e59-fed9-4cc8-9e48-403f02f0c40e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5e7e9e-66b9694f59-5rtsh" Nov 02 23:55:14.347538 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:14.347448 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2tqh\" (UniqueName: \"kubernetes.io/projected/432f3e59-fed9-4cc8-9e48-403f02f0c40e-kube-api-access-f2tqh\") pod \"payload-pre-processing-e2e-extmod-b-5e7e9e-66b9694f59-5rtsh\" (UID: \"432f3e59-fed9-4cc8-9e48-403f02f0c40e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5e7e9e-66b9694f59-5rtsh" Nov 02 23:55:14.387990 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:14.387957 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c07885-2e05-4f96-b72d-2e2a3d918bc5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-5e7e9e-6cc6b59469-q88rm\" (UID: \"05c07885-2e05-4f96-b72d-2e2a3d918bc5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5e7e9e-6cc6b59469-q88rm" Nov 02 23:55:14.388192 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:14.388169 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wmch\" (UniqueName: \"kubernetes.io/projected/05c07885-2e05-4f96-b72d-2e2a3d918bc5-kube-api-access-7wmch\") pod \"payload-processing-e2e-extmod-b-5e7e9e-6cc6b59469-q88rm\" (UID: \"05c07885-2e05-4f96-b72d-2e2a3d918bc5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5e7e9e-6cc6b59469-q88rm" Nov 02 23:55:14.388774 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:14.388743 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c07885-2e05-4f96-b72d-2e2a3d918bc5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-5e7e9e-6cc6b59469-q88rm\" (UID: \"05c07885-2e05-4f96-b72d-2e2a3d918bc5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5e7e9e-6cc6b59469-q88rm" Nov 02 23:55:14.389479 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:14.389456 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5e7e9e-66b9694f59-5rtsh" Nov 02 23:55:14.452311 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:14.450315 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wmch\" (UniqueName: \"kubernetes.io/projected/05c07885-2e05-4f96-b72d-2e2a3d918bc5-kube-api-access-7wmch\") pod \"payload-processing-e2e-extmod-b-5e7e9e-6cc6b59469-q88rm\" (UID: \"05c07885-2e05-4f96-b72d-2e2a3d918bc5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5e7e9e-6cc6b59469-q88rm" Nov 02 23:55:14.539608 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:14.539579 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-5e7e9e-6cc6b59469-q88rm" Nov 02 23:55:14.772858 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:14.772828 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5e7e9e-66b9694f59-5rtsh"] Nov 02 23:55:14.773674 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:55:14.773643 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432f3e59_fed9_4cc8_9e48_403f02f0c40e.slice/crio-f1a92b68ad656519db77fdd11167e19dd958eebd1191347c2be899c3bea5570b WatchSource:0}: Error finding container f1a92b68ad656519db77fdd11167e19dd958eebd1191347c2be899c3bea5570b: Status 404 returned error can't find the container with id f1a92b68ad656519db77fdd11167e19dd958eebd1191347c2be899c3bea5570b Nov 02 23:55:14.897185 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:14.897121 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5e7e9e-66b9694f59-5rtsh" event={"ID":"432f3e59-fed9-4cc8-9e48-403f02f0c40e","Type":"ContainerStarted","Data":"b566d3df62728bcac0bab3091d6fed53d64da29aa5203a04d5101e626cc4e627"} Nov 02 23:55:14.897384 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:14.897218 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5e7e9e-66b9694f59-5rtsh" event={"ID":"432f3e59-fed9-4cc8-9e48-403f02f0c40e","Type":"ContainerStarted","Data":"f1a92b68ad656519db77fdd11167e19dd958eebd1191347c2be899c3bea5570b"} Nov 02 23:55:14.898244 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:14.898220 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5e7e9e-66b9694f59-5rtsh" Nov 02 23:55:14.932307 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:14.932276 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5e7e9e-6cc6b59469-q88rm"] Nov 02 23:55:14.932932 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:55:14.932911 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c07885_2e05_4f96_b72d_2e2a3d918bc5.slice/crio-1b656ea22fb907b7cbd97e5f0509daa265caee6630c33bb8fe36726f0333cb32 WatchSource:0}: Error finding container 1b656ea22fb907b7cbd97e5f0509daa265caee6630c33bb8fe36726f0333cb32: Status 404 returned error can't find the container with id 1b656ea22fb907b7cbd97e5f0509daa265caee6630c33bb8fe36726f0333cb32 Nov 02 23:55:15.020288 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:15.019571 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5e7e9e-66b9694f59-5rtsh" podStartSLOduration=1.019551536 podStartE2EDuration="1.019551536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:55:15.017325785 +0000 UTC m=+4001.255823450" watchObservedRunningTime="2026-11-02 23:55:15.019551536 +0000 UTC m=+4001.258049201" Nov 02 23:55:15.905724 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:15.905683 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-5e7e9e-6cc6b59469-q88rm" event={"ID":"05c07885-2e05-4f96-b72d-2e2a3d918bc5","Type":"ContainerStarted","Data":"355ab65ce59c6d1f761e12aca4384e5b5aa740180f5ab69c5e873e952985e91d"} Nov 02 23:55:15.906223 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:15.905738 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-5e7e9e-6cc6b59469-q88rm" event={"ID":"05c07885-2e05-4f96-b72d-2e2a3d918bc5","Type":"ContainerStarted","Data":"1b656ea22fb907b7cbd97e5f0509daa265caee6630c33bb8fe36726f0333cb32"} Nov 02 23:55:15.906223 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:15.905764 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-5e7e9e-6cc6b59469-q88rm" Nov 02 23:55:16.025462 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:16.025400 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-5e7e9e-6cc6b59469-q88rm" podStartSLOduration=2.025382789 podStartE2EDuration="2.025382789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:55:16.023445813 +0000 UTC m=+4002.261943496" watchObservedRunningTime="2026-11-02 23:55:16.025382789 +0000 UTC m=+4002.263880456" Nov 02 23:55:22.302244 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:22.302206 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7097dd-54699f6fdc-99kgm"] Nov 02 23:55:22.307626 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:22.307605 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7097dd-54699f6fdc-99kgm" Nov 02 23:55:22.362522 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:22.362484 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7097dd-54699f6fdc-99kgm"] Nov 02 23:55:22.386483 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:22.386416 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7097dd\"" Nov 02 23:55:22.405843 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:22.405807 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b42mb\" (UniqueName: \"kubernetes.io/projected/830b5c2a-dd4e-4252-b9e8-c9afe2fbda88-kube-api-access-b42mb\") pod \"payload-pre-processing-e2e-inf-a-7097dd-54699f6fdc-99kgm\" (UID: \"830b5c2a-dd4e-4252-b9e8-c9afe2fbda88\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7097dd-54699f6fdc-99kgm" Nov 02 23:55:22.406047 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:22.405950 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/830b5c2a-dd4e-4252-b9e8-c9afe2fbda88-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7097dd-54699f6fdc-99kgm\" (UID: \"830b5c2a-dd4e-4252-b9e8-c9afe2fbda88\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7097dd-54699f6fdc-99kgm" Nov 02 23:55:22.407386 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:22.407359 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7097dd-dockercfg-z6z5j\"" Nov 02 23:55:22.457528 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:22.457497 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7097dd-74ff8666cf-9vbkm"] Nov 02 23:55:22.462081 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:22.462063 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7097dd-74ff8666cf-9vbkm" Nov 02 23:55:22.507049 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:22.507013 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b42mb\" (UniqueName: \"kubernetes.io/projected/830b5c2a-dd4e-4252-b9e8-c9afe2fbda88-kube-api-access-b42mb\") pod \"payload-pre-processing-e2e-inf-a-7097dd-54699f6fdc-99kgm\" (UID: \"830b5c2a-dd4e-4252-b9e8-c9afe2fbda88\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7097dd-54699f6fdc-99kgm" Nov 02 23:55:22.507276 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:22.507099 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqjq4\" (UniqueName: \"kubernetes.io/projected/0751000b-140a-4c61-906e-0c39dbddfef9-kube-api-access-tqjq4\") pod \"payload-processing-e2e-inf-a-7097dd-74ff8666cf-9vbkm\" (UID: \"0751000b-140a-4c61-906e-0c39dbddfef9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7097dd-74ff8666cf-9vbkm" Nov 02 23:55:22.507276 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:22.507256 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0751000b-140a-4c61-906e-0c39dbddfef9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7097dd-74ff8666cf-9vbkm\" (UID: \"0751000b-140a-4c61-906e-0c39dbddfef9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7097dd-74ff8666cf-9vbkm" Nov 02 23:55:22.507418 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:22.507297 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/830b5c2a-dd4e-4252-b9e8-c9afe2fbda88-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7097dd-54699f6fdc-99kgm\" (UID: \"830b5c2a-dd4e-4252-b9e8-c9afe2fbda88\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7097dd-54699f6fdc-99kgm" Nov 02 23:55:22.508002 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:22.507977 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/830b5c2a-dd4e-4252-b9e8-c9afe2fbda88-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7097dd-54699f6fdc-99kgm\" (UID: \"830b5c2a-dd4e-4252-b9e8-c9afe2fbda88\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7097dd-54699f6fdc-99kgm" Nov 02 23:55:22.518821 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:22.518795 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7097dd-74ff8666cf-9vbkm"] Nov 02 23:55:22.568275 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:22.568247 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b42mb\" (UniqueName: \"kubernetes.io/projected/830b5c2a-dd4e-4252-b9e8-c9afe2fbda88-kube-api-access-b42mb\") pod \"payload-pre-processing-e2e-inf-a-7097dd-54699f6fdc-99kgm\" (UID: \"830b5c2a-dd4e-4252-b9e8-c9afe2fbda88\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7097dd-54699f6fdc-99kgm" Nov 02 23:55:22.608531 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:22.608497 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0751000b-140a-4c61-906e-0c39dbddfef9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7097dd-74ff8666cf-9vbkm\" (UID: \"0751000b-140a-4c61-906e-0c39dbddfef9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7097dd-74ff8666cf-9vbkm" Nov 02 23:55:22.608678 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:22.608653 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqjq4\" (UniqueName: \"kubernetes.io/projected/0751000b-140a-4c61-906e-0c39dbddfef9-kube-api-access-tqjq4\") pod \"payload-processing-e2e-inf-a-7097dd-74ff8666cf-9vbkm\" (UID: \"0751000b-140a-4c61-906e-0c39dbddfef9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7097dd-74ff8666cf-9vbkm" Nov 02 23:55:22.609256 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:22.609237 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0751000b-140a-4c61-906e-0c39dbddfef9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7097dd-74ff8666cf-9vbkm\" (UID: \"0751000b-140a-4c61-906e-0c39dbddfef9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7097dd-74ff8666cf-9vbkm" Nov 02 23:55:22.622820 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:22.622802 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7097dd-54699f6fdc-99kgm" Nov 02 23:55:22.670078 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:22.670016 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqjq4\" (UniqueName: \"kubernetes.io/projected/0751000b-140a-4c61-906e-0c39dbddfef9-kube-api-access-tqjq4\") pod \"payload-processing-e2e-inf-a-7097dd-74ff8666cf-9vbkm\" (UID: \"0751000b-140a-4c61-906e-0c39dbddfef9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7097dd-74ff8666cf-9vbkm" Nov 02 23:55:22.781106 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:22.781071 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7097dd-74ff8666cf-9vbkm" Nov 02 23:55:23.010083 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:23.010053 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7097dd-54699f6fdc-99kgm"] Nov 02 23:55:23.010405 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:55:23.010376 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod830b5c2a_dd4e_4252_b9e8_c9afe2fbda88.slice/crio-8782211ec14eb829959fbb9e9ed1d00df630c41194d9639568b647c66ce8513b WatchSource:0}: Error finding container 8782211ec14eb829959fbb9e9ed1d00df630c41194d9639568b647c66ce8513b: Status 404 returned error can't find the container with id 8782211ec14eb829959fbb9e9ed1d00df630c41194d9639568b647c66ce8513b Nov 02 23:55:23.162654 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:23.162624 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7097dd-74ff8666cf-9vbkm"] Nov 02 23:55:23.164230 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:55:23.164197 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0751000b_140a_4c61_906e_0c39dbddfef9.slice/crio-4bfa3fc6d326d2d43d77152e0658a8a3f5485a63df0c2f5b69841ea8950efcf1 WatchSource:0}: Error finding container 4bfa3fc6d326d2d43d77152e0658a8a3f5485a63df0c2f5b69841ea8950efcf1: Status 404 returned error can't find the container with id 4bfa3fc6d326d2d43d77152e0658a8a3f5485a63df0c2f5b69841ea8950efcf1 Nov 02 23:55:23.970505 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:23.970465 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7097dd-74ff8666cf-9vbkm" event={"ID":"0751000b-140a-4c61-906e-0c39dbddfef9","Type":"ContainerStarted","Data":"2f2058f826925cd0d9c0291cc3ca15fa114d34505d9c0705f5926353f18cd74d"} Nov 02 23:55:23.971000 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:23.970534 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7097dd-74ff8666cf-9vbkm" event={"ID":"0751000b-140a-4c61-906e-0c39dbddfef9","Type":"ContainerStarted","Data":"4bfa3fc6d326d2d43d77152e0658a8a3f5485a63df0c2f5b69841ea8950efcf1"} Nov 02 23:55:23.971000 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:23.970586 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7097dd-74ff8666cf-9vbkm" Nov 02 23:55:23.972468 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:23.972438 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7097dd-54699f6fdc-99kgm" event={"ID":"830b5c2a-dd4e-4252-b9e8-c9afe2fbda88","Type":"ContainerStarted","Data":"9c53bb2c803a401b3fc016949aa5a44c584bf7e64c6fb220bdbfac7ad90afc92"} Nov 02 23:55:23.972611 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:23.972484 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7097dd-54699f6fdc-99kgm" Nov 02 23:55:23.972611 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:23.972501 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7097dd-54699f6fdc-99kgm" event={"ID":"830b5c2a-dd4e-4252-b9e8-c9afe2fbda88","Type":"ContainerStarted","Data":"8782211ec14eb829959fbb9e9ed1d00df630c41194d9639568b647c66ce8513b"} Nov 02 23:55:24.088188 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:24.088117 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-7097dd-74ff8666cf-9vbkm" podStartSLOduration=2.088102859 podStartE2EDuration="2.088102859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:55:24.085679248 +0000 UTC m=+4010.324176912" watchObservedRunningTime="2026-11-02 23:55:24.088102859 +0000 UTC m=+4010.326600521" Nov 02 23:55:24.205158 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:24.205071 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7097dd-54699f6fdc-99kgm" podStartSLOduration=2.205055738 podStartE2EDuration="2.205055738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:55:24.200231514 +0000 UTC m=+4010.438729199" watchObservedRunningTime="2026-11-02 23:55:24.205055738 +0000 UTC m=+4010.443553401" Nov 02 23:55:26.916889 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:26.916860 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-5e7e9e-6cc6b59469-q88rm" Nov 02 23:55:26.917303 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:26.916933 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5e7e9e-66b9694f59-5rtsh" Nov 02 23:55:27.846293 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:55:27.846253 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:55:34.984358 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:34.984328 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7097dd-54699f6fdc-99kgm" Nov 02 23:55:34.984847 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:34.984399 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7097dd-74ff8666cf-9vbkm" Nov 02 23:55:35.319107 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:35.319074 5727 scope.go:122] "RemoveContainer" containerID="8c8f45c74d3711a5d7531cdb0a19708ff11076d33dd139021e2e7661e506d591" Nov 02 23:55:44.617949 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:44.617913 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn"] Nov 02 23:55:44.622449 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:44.622429 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:44.696427 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:44.696392 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js2bw\" (UniqueName: \"kubernetes.io/projected/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-kube-api-access-js2bw\") pod \"maas-default-gateway-openshift-default-694db55c87-svghn\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:44.696618 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:44.696449 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-svghn\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:44.696618 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:44.696547 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-svghn\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:44.696618 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:44.696604 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-svghn\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:44.696813 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:44.696716 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-svghn\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:44.696873 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:44.696808 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-svghn\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:44.696873 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:44.696851 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-svghn\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:44.696990 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:44.696958 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-svghn\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:44.697042 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:44.696992 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-svghn\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:44.728571 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:44.728542 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn"] Nov 02 23:55:44.798482 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:44.798441 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-svghn\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:44.798653 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:44.798534 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js2bw\" (UniqueName: \"kubernetes.io/projected/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-kube-api-access-js2bw\") pod \"maas-default-gateway-openshift-default-694db55c87-svghn\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:44.798653 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:44.798581 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-svghn\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:44.798653 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:44.798634 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-svghn\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:44.798841 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:44.798779 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-svghn\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:44.798939 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:44.798856 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-svghn\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:44.798939 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:44.798925 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-svghn\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:44.799057 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:44.799022 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-svghn\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:44.799057 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:44.799029 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-svghn\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:44.799193 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:44.799081 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-svghn\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:44.799193 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:44.799172 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-svghn\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:44.799315 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:44.799171 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-svghn\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:44.799464 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:44.799428 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-svghn\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:44.799984 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:44.799955 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-svghn\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:44.801354 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:44.801330 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-svghn\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:44.801624 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:44.801603 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-svghn\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:44.859183 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:44.859118 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-svghn\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:44.859407 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:44.859388 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js2bw\" (UniqueName: \"kubernetes.io/projected/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-kube-api-access-js2bw\") pod \"maas-default-gateway-openshift-default-694db55c87-svghn\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:44.936907 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:44.936843 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:45.382675 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:45.382346 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn"] Nov 02 23:55:45.389479 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:45.389438 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:55:45.389609 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:45.389526 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:55:45.389609 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:45.389569 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 02 23:55:46.189992 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:46.189955 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" event={"ID":"71f62bea-0cbe-4c6d-9cd3-63893df09a9e","Type":"ContainerStarted","Data":"2330b5cfa7071fa6edcd2570bf3c36869c20f61c8671085aaaf7ef0616870735"} Nov 02 23:55:46.190402 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:46.190295 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" event={"ID":"71f62bea-0cbe-4c6d-9cd3-63893df09a9e","Type":"ContainerStarted","Data":"d64cf8bb0df2fb86aca6723f681f079fd813f77d775a186dc10ede90b6a56b0d"} Nov 02 23:55:46.544238 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:46.544142 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" podStartSLOduration=2.54411485 podStartE2EDuration="2.54411485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:55:46.368845883 +0000 UTC m=+4032.607343569" watchObservedRunningTime="2026-11-02 23:55:46.54411485 +0000 UTC m=+4032.782612511" Nov 02 23:55:46.545806 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:46.545786 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h"] Nov 02 23:55:46.571154 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:46.560404 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h" Nov 02 23:55:46.609842 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:46.609809 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h"] Nov 02 23:55:46.612847 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:46.612824 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-18a6d8-serving-cert\"" Nov 02 23:55:46.622666 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:46.622629 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/660773e5-8de6-423d-9718-71742230175a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h\" (UID: \"660773e5-8de6-423d-9718-71742230175a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h" Nov 02 23:55:46.622934 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:46.622908 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/660773e5-8de6-423d-9718-71742230175a-maas-api-tls\") pod \"maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h\" (UID: \"660773e5-8de6-423d-9718-71742230175a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h" Nov 02 23:55:46.623042 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:46.623023 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/660773e5-8de6-423d-9718-71742230175a-metrics-certs\") pod \"maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h\" (UID: \"660773e5-8de6-423d-9718-71742230175a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h" Nov 02 23:55:46.623087 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:46.623074 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg52q\" (UniqueName: \"kubernetes.io/projected/660773e5-8de6-423d-9718-71742230175a-kube-api-access-sg52q\") pod \"maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h\" (UID: \"660773e5-8de6-423d-9718-71742230175a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h" Nov 02 23:55:46.723990 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:46.723949 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/660773e5-8de6-423d-9718-71742230175a-maas-api-tls\") pod \"maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h\" (UID: \"660773e5-8de6-423d-9718-71742230175a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h" Nov 02 23:55:46.724195 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:46.724037 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/660773e5-8de6-423d-9718-71742230175a-metrics-certs\") pod \"maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h\" (UID: \"660773e5-8de6-423d-9718-71742230175a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h" Nov 02 23:55:46.724195 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:46.724093 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg52q\" (UniqueName: \"kubernetes.io/projected/660773e5-8de6-423d-9718-71742230175a-kube-api-access-sg52q\") pod \"maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h\" (UID: \"660773e5-8de6-423d-9718-71742230175a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h" Nov 02 23:55:46.724195 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:55:46.724109 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-18a6d8-serving-cert: secret "maas-api-e2e-inf-a-18a6d8-serving-cert" not found Nov 02 23:55:46.724351 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:55:46.724200 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/660773e5-8de6-423d-9718-71742230175a-maas-api-tls podName:660773e5-8de6-423d-9718-71742230175a nodeName:}" failed. No retries permitted until 2026-11-02 23:55:47.224177874 +0000 UTC m=+4033.462675534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/660773e5-8de6-423d-9718-71742230175a-maas-api-tls") pod "maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h" (UID: "660773e5-8de6-423d-9718-71742230175a") : secret "maas-api-e2e-inf-a-18a6d8-serving-cert" not found Nov 02 23:55:46.724351 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:46.724256 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/660773e5-8de6-423d-9718-71742230175a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h\" (UID: \"660773e5-8de6-423d-9718-71742230175a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h" Nov 02 23:55:46.725067 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:46.725038 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/660773e5-8de6-423d-9718-71742230175a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h\" (UID: \"660773e5-8de6-423d-9718-71742230175a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h" Nov 02 23:55:46.727393 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:46.727371 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/660773e5-8de6-423d-9718-71742230175a-metrics-certs\") pod \"maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h\" (UID: \"660773e5-8de6-423d-9718-71742230175a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h" Nov 02 23:55:46.797016 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:46.796259 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg52q\" (UniqueName: \"kubernetes.io/projected/660773e5-8de6-423d-9718-71742230175a-kube-api-access-sg52q\") pod \"maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h\" (UID: \"660773e5-8de6-423d-9718-71742230175a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h" Nov 02 23:55:46.937353 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:46.937320 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:46.938982 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:46.938961 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-svghn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Nov 02 23:55:46.939100 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:46.939013 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" podUID="71f62bea-0cbe-4c6d-9cd3-63893df09a9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Nov 02 23:55:47.231879 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:47.231842 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/660773e5-8de6-423d-9718-71742230175a-maas-api-tls\") pod \"maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h\" (UID: \"660773e5-8de6-423d-9718-71742230175a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h" Nov 02 23:55:47.234518 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:47.234498 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/660773e5-8de6-423d-9718-71742230175a-maas-api-tls\") pod \"maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h\" (UID: \"660773e5-8de6-423d-9718-71742230175a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h" Nov 02 23:55:47.500980 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:47.500888 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h" Nov 02 23:55:47.936356 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:47.936325 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h"] Nov 02 23:55:47.937495 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:47.937476 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-svghn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Nov 02 23:55:47.937602 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:47.937517 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" podUID="71f62bea-0cbe-4c6d-9cd3-63893df09a9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Nov 02 23:55:47.937844 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:55:47.937821 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660773e5_8de6_423d_9718_71742230175a.slice/crio-fb70efad19e0563572f81e81bca7f8b2e8c9edab250bf1f6db66fcf9862b9ea2 WatchSource:0}: Error finding container fb70efad19e0563572f81e81bca7f8b2e8c9edab250bf1f6db66fcf9862b9ea2: Status 404 returned error can't find the container with id fb70efad19e0563572f81e81bca7f8b2e8c9edab250bf1f6db66fcf9862b9ea2 Nov 02 23:55:48.211007 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:48.210915 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h" event={"ID":"660773e5-8de6-423d-9718-71742230175a","Type":"ContainerStarted","Data":"fb70efad19e0563572f81e81bca7f8b2e8c9edab250bf1f6db66fcf9862b9ea2"} Nov 02 23:55:48.938150 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:48.937931 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-svghn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Nov 02 23:55:48.938150 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:48.937990 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" podUID="71f62bea-0cbe-4c6d-9cd3-63893df09a9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Nov 02 23:55:49.222441 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:49.222353 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h" event={"ID":"660773e5-8de6-423d-9718-71742230175a","Type":"ContainerStarted","Data":"c79f68826ceb9b9b937d9987e6bf2b5718e46be01bc0dc3f0e0b8ed75e0cc115"} Nov 02 23:55:49.222597 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:49.222441 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h" Nov 02 23:55:49.341118 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:49.341065 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h" podStartSLOduration=2.463151403 podStartE2EDuration="3.341050456s" totalImagesPullingTime="877.899053ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:55:49.339654983 +0000 UTC m=+4035.578152646" watchObservedRunningTime="2026-11-02 23:55:49.341050456 +0000 UTC m=+4035.579548118" Nov 02 23:55:49.937320 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:49.937286 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-svghn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Nov 02 23:55:49.937504 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:49.937345 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" podUID="71f62bea-0cbe-4c6d-9cd3-63893df09a9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Nov 02 23:55:50.937796 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:50.937754 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-svghn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Nov 02 23:55:50.938210 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:50.937816 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" podUID="71f62bea-0cbe-4c6d-9cd3-63893df09a9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Nov 02 23:55:51.938012 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:51.937978 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-svghn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Nov 02 23:55:51.938427 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:51.938044 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" podUID="71f62bea-0cbe-4c6d-9cd3-63893df09a9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Nov 02 23:55:52.938007 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:52.937961 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-svghn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Nov 02 23:55:52.938231 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:52.938039 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" podUID="71f62bea-0cbe-4c6d-9cd3-63893df09a9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Nov 02 23:55:54.126818 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:54.126788 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:54.127340 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:54.127300 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:54.128274 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:54.128249 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 02 23:55:55.236654 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:55:55.236621 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h" Nov 02 23:55:57.850712 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:55:57.850672 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:56:27.847307 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:56:27.847259 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:56:51.687689 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:56:51.687656 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2"] Nov 02 23:56:51.692336 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:56:51.692317 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2" Nov 02 23:56:51.795239 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:56:51.795208 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7097dd-serving-cert\"" Nov 02 23:56:51.800562 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:56:51.800540 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2"] Nov 02 23:56:51.819764 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:56:51.819713 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12cd1a89-fec7-4b82-b481-1022d5cbdd32-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2\" (UID: \"12cd1a89-fec7-4b82-b481-1022d5cbdd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2" Nov 02 23:56:51.819947 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:56:51.819787 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12cd1a89-fec7-4b82-b481-1022d5cbdd32-metrics-certs\") pod \"maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2\" (UID: \"12cd1a89-fec7-4b82-b481-1022d5cbdd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2" Nov 02 23:56:51.820088 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:56:51.820068 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sscr5\" (UniqueName: \"kubernetes.io/projected/12cd1a89-fec7-4b82-b481-1022d5cbdd32-kube-api-access-sscr5\") pod \"maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2\" (UID: \"12cd1a89-fec7-4b82-b481-1022d5cbdd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2" Nov 02 23:56:51.820205 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:56:51.820107 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12cd1a89-fec7-4b82-b481-1022d5cbdd32-maas-api-tls\") pod \"maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2\" (UID: \"12cd1a89-fec7-4b82-b481-1022d5cbdd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2" Nov 02 23:56:51.921090 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:56:51.921052 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12cd1a89-fec7-4b82-b481-1022d5cbdd32-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2\" (UID: \"12cd1a89-fec7-4b82-b481-1022d5cbdd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2" Nov 02 23:56:51.921090 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:56:51.921087 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12cd1a89-fec7-4b82-b481-1022d5cbdd32-metrics-certs\") pod \"maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2\" (UID: \"12cd1a89-fec7-4b82-b481-1022d5cbdd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2" Nov 02 23:56:51.921382 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:56:51.921249 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sscr5\" (UniqueName: \"kubernetes.io/projected/12cd1a89-fec7-4b82-b481-1022d5cbdd32-kube-api-access-sscr5\") pod \"maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2\" (UID: \"12cd1a89-fec7-4b82-b481-1022d5cbdd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2" Nov 02 23:56:51.921382 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:56:51.921279 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12cd1a89-fec7-4b82-b481-1022d5cbdd32-maas-api-tls\") pod \"maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2\" (UID: \"12cd1a89-fec7-4b82-b481-1022d5cbdd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2" Nov 02 23:56:51.921501 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:56:51.921411 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-7097dd-serving-cert: secret "maas-api-e2e-inf-a-7097dd-serving-cert" not found Nov 02 23:56:51.921501 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:56:51.921475 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12cd1a89-fec7-4b82-b481-1022d5cbdd32-maas-api-tls podName:12cd1a89-fec7-4b82-b481-1022d5cbdd32 nodeName:}" failed. No retries permitted until 2026-11-02 23:56:52.421457598 +0000 UTC m=+4098.659955253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/12cd1a89-fec7-4b82-b481-1022d5cbdd32-maas-api-tls") pod "maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2" (UID: "12cd1a89-fec7-4b82-b481-1022d5cbdd32") : secret "maas-api-e2e-inf-a-7097dd-serving-cert" not found Nov 02 23:56:51.921810 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:56:51.921784 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12cd1a89-fec7-4b82-b481-1022d5cbdd32-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2\" (UID: \"12cd1a89-fec7-4b82-b481-1022d5cbdd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2" Nov 02 23:56:51.923986 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:56:51.923962 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12cd1a89-fec7-4b82-b481-1022d5cbdd32-metrics-certs\") pod \"maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2\" (UID: \"12cd1a89-fec7-4b82-b481-1022d5cbdd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2" Nov 02 23:56:51.985034 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:56:51.984940 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sscr5\" (UniqueName: \"kubernetes.io/projected/12cd1a89-fec7-4b82-b481-1022d5cbdd32-kube-api-access-sscr5\") pod \"maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2\" (UID: \"12cd1a89-fec7-4b82-b481-1022d5cbdd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2" Nov 02 23:56:52.427268 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:56:52.427233 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12cd1a89-fec7-4b82-b481-1022d5cbdd32-maas-api-tls\") pod \"maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2\" (UID: \"12cd1a89-fec7-4b82-b481-1022d5cbdd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2" Nov 02 23:56:52.430013 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:56:52.429986 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12cd1a89-fec7-4b82-b481-1022d5cbdd32-maas-api-tls\") pod \"maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2\" (UID: \"12cd1a89-fec7-4b82-b481-1022d5cbdd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2" Nov 02 23:56:52.607792 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:56:52.607757 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2" Nov 02 23:56:53.010455 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:56:53.010426 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2"] Nov 02 23:56:53.011234 ip-10-0-135-176 kubenswrapper[5727]: W1102 23:56:53.011210 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12cd1a89_fec7_4b82_b481_1022d5cbdd32.slice/crio-8e2f85832a05aa0dbb169f98ada70c4a1a24b01e46ca91eedb32a8e8948e1135 WatchSource:0}: Error finding container 8e2f85832a05aa0dbb169f98ada70c4a1a24b01e46ca91eedb32a8e8948e1135: Status 404 returned error can't find the container with id 8e2f85832a05aa0dbb169f98ada70c4a1a24b01e46ca91eedb32a8e8948e1135 Nov 02 23:56:53.752967 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:56:53.752926 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2" event={"ID":"12cd1a89-fec7-4b82-b481-1022d5cbdd32","Type":"ContainerStarted","Data":"8e2f85832a05aa0dbb169f98ada70c4a1a24b01e46ca91eedb32a8e8948e1135"} Nov 02 23:56:54.766325 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:56:54.766288 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2" event={"ID":"12cd1a89-fec7-4b82-b481-1022d5cbdd32","Type":"ContainerStarted","Data":"f45d54e68b3a413120cc491d8b3b8e03936fedb69a04e48ccd877d96d4bc6bd9"} Nov 02 23:56:54.766866 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:56:54.766385 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2" Nov 02 23:56:54.885631 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:56:54.885572 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2" podStartSLOduration=3.031070314 podStartE2EDuration="3.885555977s" totalImagesPullingTime="854.485663ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:56:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:56:54.882760362 +0000 UTC m=+4101.121258049" watchObservedRunningTime="2026-11-02 23:56:54.885555977 +0000 UTC m=+4101.124053638" Nov 02 23:56:57.846876 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:56:57.846836 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:57:00.782711 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:00.782682 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2" Nov 02 23:57:27.847663 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:57:27.847625 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:57:37.329352 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:37.329288 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-18a6d8-55c5858669-hts8r"] Nov 02 23:57:37.435469 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:37.435423 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-18a6d8-667d88ff5-fg8ng"] Nov 02 23:57:42.330469 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:42.330404 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-18a6d8-55c5858669-hts8r" podUID="78557250-4ce3-404d-b057-e66bde4fec4d" containerName="payload-processing" containerID="cri-o://67d22fd482365be5fd05bbb22c120ed0775825ac7e6f23019f5ad79bdecfcb80" gracePeriod=25 Nov 02 23:57:42.437102 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:42.437057 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18a6d8-667d88ff5-fg8ng" podUID="aa393900-e35d-4751-817c-c6ed6a2d6a8a" containerName="payload-pre-processing" containerID="cri-o://7a920323cd722507ecbaf8991f8e3a3eda8bcb225976560c6e3185e134293f6d" gracePeriod=25 Nov 02 23:57:42.628708 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:42.628680 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-18a6d8-55c5858669-hts8r" Nov 02 23:57:42.665054 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:42.665001 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78557250-4ce3-404d-b057-e66bde4fec4d-kube-api-access-7fkdc\" (UniqueName: \"kubernetes.io/projected/78557250-4ce3-404d-b057-e66bde4fec4d-kube-api-access-7fkdc\") pod \"78557250-4ce3-404d-b057-e66bde4fec4d\" (UID: \"78557250-4ce3-404d-b057-e66bde4fec4d\") " Nov 02 23:57:42.665256 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:42.665116 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78557250-4ce3-404d-b057-e66bde4fec4d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78557250-4ce3-404d-b057-e66bde4fec4d-plugins-config-volume\") pod \"78557250-4ce3-404d-b057-e66bde4fec4d\" (UID: \"78557250-4ce3-404d-b057-e66bde4fec4d\") " Nov 02 23:57:42.672180 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:42.665536 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78557250-4ce3-404d-b057-e66bde4fec4d-plugins-config-volume" pod "78557250-4ce3-404d-b057-e66bde4fec4d" (UID: "78557250-4ce3-404d-b057-e66bde4fec4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:57:42.672180 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:42.665719 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78557250-4ce3-404d-b057-e66bde4fec4d-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:57:42.672180 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:42.668241 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78557250-4ce3-404d-b057-e66bde4fec4d-kube-api-access-7fkdc" pod "78557250-4ce3-404d-b057-e66bde4fec4d" (UID: "78557250-4ce3-404d-b057-e66bde4fec4d"). InnerVolumeSpecName "kube-api-access-7fkdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:57:42.740165 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:42.740101 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18a6d8-667d88ff5-fg8ng" Nov 02 23:57:42.767308 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:42.766836 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fkdc\" (UniqueName: \"kubernetes.io/projected/78557250-4ce3-404d-b057-e66bde4fec4d-kube-api-access-7fkdc\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:57:42.868718 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:42.868638 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa393900-e35d-4751-817c-c6ed6a2d6a8a-kube-api-access-s4ft2\" (UniqueName: \"kubernetes.io/projected/aa393900-e35d-4751-817c-c6ed6a2d6a8a-kube-api-access-s4ft2\") pod \"aa393900-e35d-4751-817c-c6ed6a2d6a8a\" (UID: \"aa393900-e35d-4751-817c-c6ed6a2d6a8a\") " Nov 02 23:57:42.868718 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:42.868679 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa393900-e35d-4751-817c-c6ed6a2d6a8a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa393900-e35d-4751-817c-c6ed6a2d6a8a-plugins-config-volume\") pod \"aa393900-e35d-4751-817c-c6ed6a2d6a8a\" (UID: \"aa393900-e35d-4751-817c-c6ed6a2d6a8a\") " Nov 02 23:57:42.869107 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:42.869079 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa393900-e35d-4751-817c-c6ed6a2d6a8a-plugins-config-volume" pod "aa393900-e35d-4751-817c-c6ed6a2d6a8a" (UID: "aa393900-e35d-4751-817c-c6ed6a2d6a8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:57:42.869298 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:42.869278 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa393900-e35d-4751-817c-c6ed6a2d6a8a-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:57:42.871079 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:42.871054 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa393900-e35d-4751-817c-c6ed6a2d6a8a-kube-api-access-s4ft2" pod "aa393900-e35d-4751-817c-c6ed6a2d6a8a" (UID: "aa393900-e35d-4751-817c-c6ed6a2d6a8a"). InnerVolumeSpecName "kube-api-access-s4ft2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:57:42.970338 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:42.970297 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4ft2\" (UniqueName: \"kubernetes.io/projected/aa393900-e35d-4751-817c-c6ed6a2d6a8a-kube-api-access-s4ft2\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:57:43.154678 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:43.154593 5727 generic.go:356] "Generic (PLEG): container finished" podID="78557250-4ce3-404d-b057-e66bde4fec4d" containerID="67d22fd482365be5fd05bbb22c120ed0775825ac7e6f23019f5ad79bdecfcb80" exitCode=0 Nov 02 23:57:43.154678 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:43.154639 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-18a6d8-55c5858669-hts8r" Nov 02 23:57:43.154925 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:43.154681 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-18a6d8-55c5858669-hts8r" event={"ID":"78557250-4ce3-404d-b057-e66bde4fec4d","Type":"ContainerDied","Data":"67d22fd482365be5fd05bbb22c120ed0775825ac7e6f23019f5ad79bdecfcb80"} Nov 02 23:57:43.154925 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:43.154740 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-18a6d8-55c5858669-hts8r" event={"ID":"78557250-4ce3-404d-b057-e66bde4fec4d","Type":"ContainerDied","Data":"edf0f176d184dc90c2bc2f0e42d10436115921c9812f07e267b3740961d78006"} Nov 02 23:57:43.154925 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:43.154770 5727 scope.go:122] "RemoveContainer" containerID="67d22fd482365be5fd05bbb22c120ed0775825ac7e6f23019f5ad79bdecfcb80" Nov 02 23:57:43.156604 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:43.156578 5727 generic.go:356] "Generic (PLEG): container finished" podID="aa393900-e35d-4751-817c-c6ed6a2d6a8a" containerID="7a920323cd722507ecbaf8991f8e3a3eda8bcb225976560c6e3185e134293f6d" exitCode=0 Nov 02 23:57:43.156731 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:43.156622 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18a6d8-667d88ff5-fg8ng" Nov 02 23:57:43.156731 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:43.156661 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18a6d8-667d88ff5-fg8ng" event={"ID":"aa393900-e35d-4751-817c-c6ed6a2d6a8a","Type":"ContainerDied","Data":"7a920323cd722507ecbaf8991f8e3a3eda8bcb225976560c6e3185e134293f6d"} Nov 02 23:57:43.156731 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:43.156726 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18a6d8-667d88ff5-fg8ng" event={"ID":"aa393900-e35d-4751-817c-c6ed6a2d6a8a","Type":"ContainerDied","Data":"7204dffa0996ce315ac846012b22c42b26711fd9bab5060da8cdea363cbe0a4f"} Nov 02 23:57:43.171640 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:43.171593 5727 scope.go:122] "RemoveContainer" containerID="67d22fd482365be5fd05bbb22c120ed0775825ac7e6f23019f5ad79bdecfcb80" Nov 02 23:57:43.171919 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:57:43.171892 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67d22fd482365be5fd05bbb22c120ed0775825ac7e6f23019f5ad79bdecfcb80\": container with ID starting with 67d22fd482365be5fd05bbb22c120ed0775825ac7e6f23019f5ad79bdecfcb80 not found: ID does not exist" containerID="67d22fd482365be5fd05bbb22c120ed0775825ac7e6f23019f5ad79bdecfcb80" Nov 02 23:57:43.172020 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:43.171930 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d22fd482365be5fd05bbb22c120ed0775825ac7e6f23019f5ad79bdecfcb80"} err="failed to get container status \"67d22fd482365be5fd05bbb22c120ed0775825ac7e6f23019f5ad79bdecfcb80\": rpc error: code = NotFound desc = could not find container \"67d22fd482365be5fd05bbb22c120ed0775825ac7e6f23019f5ad79bdecfcb80\": container with ID starting with 67d22fd482365be5fd05bbb22c120ed0775825ac7e6f23019f5ad79bdecfcb80 not found: ID does not exist" Nov 02 23:57:43.172020 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:43.171949 5727 scope.go:122] "RemoveContainer" containerID="7a920323cd722507ecbaf8991f8e3a3eda8bcb225976560c6e3185e134293f6d" Nov 02 23:57:43.183446 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:43.183419 5727 scope.go:122] "RemoveContainer" containerID="7a920323cd722507ecbaf8991f8e3a3eda8bcb225976560c6e3185e134293f6d" Nov 02 23:57:43.183749 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:57:43.183725 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a920323cd722507ecbaf8991f8e3a3eda8bcb225976560c6e3185e134293f6d\": container with ID starting with 7a920323cd722507ecbaf8991f8e3a3eda8bcb225976560c6e3185e134293f6d not found: ID does not exist" containerID="7a920323cd722507ecbaf8991f8e3a3eda8bcb225976560c6e3185e134293f6d" Nov 02 23:57:43.183842 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:43.183754 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a920323cd722507ecbaf8991f8e3a3eda8bcb225976560c6e3185e134293f6d"} err="failed to get container status \"7a920323cd722507ecbaf8991f8e3a3eda8bcb225976560c6e3185e134293f6d\": rpc error: code = NotFound desc = could not find container \"7a920323cd722507ecbaf8991f8e3a3eda8bcb225976560c6e3185e134293f6d\": container with ID starting with 7a920323cd722507ecbaf8991f8e3a3eda8bcb225976560c6e3185e134293f6d not found: ID does not exist" Nov 02 23:57:43.326393 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:43.326355 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-18a6d8-55c5858669-hts8r"] Nov 02 23:57:43.371894 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:43.371854 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-18a6d8-55c5858669-hts8r"] Nov 02 23:57:43.434415 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:43.434334 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-18a6d8-667d88ff5-fg8ng"] Nov 02 23:57:43.439642 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:43.439611 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-18a6d8-667d88ff5-fg8ng"] Nov 02 23:57:43.883724 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:43.883684 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78557250-4ce3-404d-b057-e66bde4fec4d" path="/var/lib/kubelet/pods/78557250-4ce3-404d-b057-e66bde4fec4d/volumes" Nov 02 23:57:43.884170 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:57:43.884148 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa393900-e35d-4751-817c-c6ed6a2d6a8a" path="/var/lib/kubelet/pods/aa393900-e35d-4751-817c-c6ed6a2d6a8a/volumes" Nov 02 23:57:57.846604 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:57:57.846564 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:58:12.423960 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:12.423926 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0f7002-6495d97797-ql9sw"] Nov 02 23:58:17.424643 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:17.424599 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f7002-6495d97797-ql9sw" podUID="e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0" containerName="payload-processing" containerID="cri-o://a12ef48c5efcf05fee561f4066f54d6fda6c87c7cf4eabb975dcbccfc3d25af8" gracePeriod=25 Nov 02 23:58:17.715333 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:17.715309 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f7002-6495d97797-ql9sw" Nov 02 23:58:17.831045 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:17.831004 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0-plugins-config-volume\") pod \"e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0\" (UID: \"e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0\") " Nov 02 23:58:17.831330 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:17.831306 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0-kube-api-access-t4lx8\" (UniqueName: \"kubernetes.io/projected/e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0-kube-api-access-t4lx8\") pod \"e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0\" (UID: \"e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0\") " Nov 02 23:58:17.831477 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:17.831452 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0-plugins-config-volume" pod "e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0" (UID: "e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:58:17.831754 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:17.831736 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:58:17.834204 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:17.834181 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0-kube-api-access-t4lx8" pod "e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0" (UID: "e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0"). InnerVolumeSpecName "kube-api-access-t4lx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:58:17.933046 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:17.932977 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4lx8\" (UniqueName: \"kubernetes.io/projected/e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0-kube-api-access-t4lx8\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:58:18.440110 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:18.440075 5727 generic.go:356] "Generic (PLEG): container finished" podID="e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0" containerID="a12ef48c5efcf05fee561f4066f54d6fda6c87c7cf4eabb975dcbccfc3d25af8" exitCode=0 Nov 02 23:58:18.440625 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:18.440153 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f7002-6495d97797-ql9sw" Nov 02 23:58:18.440625 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:18.440165 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f7002-6495d97797-ql9sw" event={"ID":"e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0","Type":"ContainerDied","Data":"a12ef48c5efcf05fee561f4066f54d6fda6c87c7cf4eabb975dcbccfc3d25af8"} Nov 02 23:58:18.440625 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:18.440254 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f7002-6495d97797-ql9sw" event={"ID":"e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0","Type":"ContainerDied","Data":"2a7375559ed2213c467b21cffa197bda651258f9aa950a980fafec59ca91a54b"} Nov 02 23:58:18.440625 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:18.440284 5727 scope.go:122] "RemoveContainer" containerID="a12ef48c5efcf05fee561f4066f54d6fda6c87c7cf4eabb975dcbccfc3d25af8" Nov 02 23:58:18.458015 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:18.457993 5727 scope.go:122] "RemoveContainer" containerID="a12ef48c5efcf05fee561f4066f54d6fda6c87c7cf4eabb975dcbccfc3d25af8" Nov 02 23:58:18.458414 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:58:18.458389 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a12ef48c5efcf05fee561f4066f54d6fda6c87c7cf4eabb975dcbccfc3d25af8\": container with ID starting with a12ef48c5efcf05fee561f4066f54d6fda6c87c7cf4eabb975dcbccfc3d25af8 not found: ID does not exist" containerID="a12ef48c5efcf05fee561f4066f54d6fda6c87c7cf4eabb975dcbccfc3d25af8" Nov 02 23:58:18.458505 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:18.458427 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a12ef48c5efcf05fee561f4066f54d6fda6c87c7cf4eabb975dcbccfc3d25af8"} err="failed to get container status \"a12ef48c5efcf05fee561f4066f54d6fda6c87c7cf4eabb975dcbccfc3d25af8\": rpc error: code = NotFound desc = could not find container \"a12ef48c5efcf05fee561f4066f54d6fda6c87c7cf4eabb975dcbccfc3d25af8\": container with ID starting with a12ef48c5efcf05fee561f4066f54d6fda6c87c7cf4eabb975dcbccfc3d25af8 not found: ID does not exist" Nov 02 23:58:18.639037 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:18.639008 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0f7002-6495d97797-ql9sw"] Nov 02 23:58:18.653789 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:18.653753 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0f7002-6495d97797-ql9sw"] Nov 02 23:58:19.886263 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:19.886206 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0" path="/var/lib/kubelet/pods/e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0/volumes" Nov 02 23:58:27.849447 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:58:27.849406 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:58:30.164434 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:30.164392 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7097dd-74ff8666cf-9vbkm"] Nov 02 23:58:30.272533 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:30.272498 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7097dd-54699f6fdc-99kgm"] Nov 02 23:58:33.672490 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:33.672449 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2"] Nov 02 23:58:33.672984 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:33.672683 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2" podUID="12cd1a89-fec7-4b82-b481-1022d5cbdd32" containerName="maas-api" containerID="cri-o://f45d54e68b3a413120cc491d8b3b8e03936fedb69a04e48ccd877d96d4bc6bd9" gracePeriod=30 Nov 02 23:58:33.983948 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:33.983921 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2" Nov 02 23:58:34.043219 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:34.043189 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12cd1a89-fec7-4b82-b481-1022d5cbdd32-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12cd1a89-fec7-4b82-b481-1022d5cbdd32-maas-api-tls\") pod \"12cd1a89-fec7-4b82-b481-1022d5cbdd32\" (UID: \"12cd1a89-fec7-4b82-b481-1022d5cbdd32\") " Nov 02 23:58:34.043377 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:34.043337 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12cd1a89-fec7-4b82-b481-1022d5cbdd32-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12cd1a89-fec7-4b82-b481-1022d5cbdd32-openshift-service-ca\") pod \"12cd1a89-fec7-4b82-b481-1022d5cbdd32\" (UID: \"12cd1a89-fec7-4b82-b481-1022d5cbdd32\") " Nov 02 23:58:34.043445 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:34.043386 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12cd1a89-fec7-4b82-b481-1022d5cbdd32-kube-api-access-sscr5\" (UniqueName: \"kubernetes.io/projected/12cd1a89-fec7-4b82-b481-1022d5cbdd32-kube-api-access-sscr5\") pod \"12cd1a89-fec7-4b82-b481-1022d5cbdd32\" (UID: \"12cd1a89-fec7-4b82-b481-1022d5cbdd32\") " Nov 02 23:58:34.043445 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:34.043406 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12cd1a89-fec7-4b82-b481-1022d5cbdd32-metrics-certs\" (UniqueName: \"kubernetes.io/secret/12cd1a89-fec7-4b82-b481-1022d5cbdd32-metrics-certs\") pod \"12cd1a89-fec7-4b82-b481-1022d5cbdd32\" (UID: \"12cd1a89-fec7-4b82-b481-1022d5cbdd32\") " Nov 02 23:58:34.043722 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:34.043694 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12cd1a89-fec7-4b82-b481-1022d5cbdd32-openshift-service-ca" pod "12cd1a89-fec7-4b82-b481-1022d5cbdd32" (UID: "12cd1a89-fec7-4b82-b481-1022d5cbdd32"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:58:34.043944 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:34.043867 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12cd1a89-fec7-4b82-b481-1022d5cbdd32-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:58:34.045550 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:34.045521 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12cd1a89-fec7-4b82-b481-1022d5cbdd32-metrics-certs" pod "12cd1a89-fec7-4b82-b481-1022d5cbdd32" (UID: "12cd1a89-fec7-4b82-b481-1022d5cbdd32"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:58:34.045636 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:34.045615 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12cd1a89-fec7-4b82-b481-1022d5cbdd32-kube-api-access-sscr5" pod "12cd1a89-fec7-4b82-b481-1022d5cbdd32" (UID: "12cd1a89-fec7-4b82-b481-1022d5cbdd32"). InnerVolumeSpecName "kube-api-access-sscr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:58:34.045677 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:34.045629 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12cd1a89-fec7-4b82-b481-1022d5cbdd32-maas-api-tls" pod "12cd1a89-fec7-4b82-b481-1022d5cbdd32" (UID: "12cd1a89-fec7-4b82-b481-1022d5cbdd32"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:58:34.144885 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:34.144851 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sscr5\" (UniqueName: \"kubernetes.io/projected/12cd1a89-fec7-4b82-b481-1022d5cbdd32-kube-api-access-sscr5\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:58:34.144885 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:34.144883 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12cd1a89-fec7-4b82-b481-1022d5cbdd32-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:58:34.145092 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:34.144896 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12cd1a89-fec7-4b82-b481-1022d5cbdd32-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:58:34.572047 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:34.572009 5727 generic.go:356] "Generic (PLEG): container finished" podID="12cd1a89-fec7-4b82-b481-1022d5cbdd32" containerID="f45d54e68b3a413120cc491d8b3b8e03936fedb69a04e48ccd877d96d4bc6bd9" exitCode=0 Nov 02 23:58:34.572277 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:34.572059 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2" Nov 02 23:58:34.572277 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:34.572080 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2" event={"ID":"12cd1a89-fec7-4b82-b481-1022d5cbdd32","Type":"ContainerDied","Data":"f45d54e68b3a413120cc491d8b3b8e03936fedb69a04e48ccd877d96d4bc6bd9"} Nov 02 23:58:34.572277 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:34.572146 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2" event={"ID":"12cd1a89-fec7-4b82-b481-1022d5cbdd32","Type":"ContainerDied","Data":"8e2f85832a05aa0dbb169f98ada70c4a1a24b01e46ca91eedb32a8e8948e1135"} Nov 02 23:58:34.572277 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:34.572175 5727 scope.go:122] "RemoveContainer" containerID="f45d54e68b3a413120cc491d8b3b8e03936fedb69a04e48ccd877d96d4bc6bd9" Nov 02 23:58:34.585795 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:34.585772 5727 scope.go:122] "RemoveContainer" containerID="f45d54e68b3a413120cc491d8b3b8e03936fedb69a04e48ccd877d96d4bc6bd9" Nov 02 23:58:34.586156 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:58:34.586108 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f45d54e68b3a413120cc491d8b3b8e03936fedb69a04e48ccd877d96d4bc6bd9\": container with ID starting with f45d54e68b3a413120cc491d8b3b8e03936fedb69a04e48ccd877d96d4bc6bd9 not found: ID does not exist" containerID="f45d54e68b3a413120cc491d8b3b8e03936fedb69a04e48ccd877d96d4bc6bd9" Nov 02 23:58:34.586238 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:34.586172 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f45d54e68b3a413120cc491d8b3b8e03936fedb69a04e48ccd877d96d4bc6bd9"} err="failed to get container status \"f45d54e68b3a413120cc491d8b3b8e03936fedb69a04e48ccd877d96d4bc6bd9\": rpc error: code = NotFound desc = could not find container \"f45d54e68b3a413120cc491d8b3b8e03936fedb69a04e48ccd877d96d4bc6bd9\": container with ID starting with f45d54e68b3a413120cc491d8b3b8e03936fedb69a04e48ccd877d96d4bc6bd9 not found: ID does not exist" Nov 02 23:58:34.737635 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:34.737601 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2"] Nov 02 23:58:34.741029 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:34.741005 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7097dd-758b4c6fff-wmkj2"] Nov 02 23:58:34.840994 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:34.840921 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 02 23:58:34.860070 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:34.860045 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 23:58:34.937511 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:34.937481 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 02 23:58:34.953320 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:34.953297 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 02 23:58:35.165000 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:35.164880 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-7097dd-74ff8666cf-9vbkm" podUID="0751000b-140a-4c61-906e-0c39dbddfef9" containerName="payload-processing" containerID="cri-o://2f2058f826925cd0d9c0291cc3ca15fa114d34505d9c0705f5926353f18cd74d" gracePeriod=25 Nov 02 23:58:35.273324 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:35.273254 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7097dd-54699f6fdc-99kgm" podUID="830b5c2a-dd4e-4252-b9e8-c9afe2fbda88" containerName="payload-pre-processing" containerID="cri-o://9c53bb2c803a401b3fc016949aa5a44c584bf7e64c6fb220bdbfac7ad90afc92" gracePeriod=25 Nov 02 23:58:35.503304 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:35.503269 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7097dd-74ff8666cf-9vbkm" Nov 02 23:58:35.566939 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:35.566901 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0751000b-140a-4c61-906e-0c39dbddfef9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0751000b-140a-4c61-906e-0c39dbddfef9-plugins-config-volume\") pod \"0751000b-140a-4c61-906e-0c39dbddfef9\" (UID: \"0751000b-140a-4c61-906e-0c39dbddfef9\") " Nov 02 23:58:35.567153 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:35.567091 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0751000b-140a-4c61-906e-0c39dbddfef9-kube-api-access-tqjq4\" (UniqueName: \"kubernetes.io/projected/0751000b-140a-4c61-906e-0c39dbddfef9-kube-api-access-tqjq4\") pod \"0751000b-140a-4c61-906e-0c39dbddfef9\" (UID: \"0751000b-140a-4c61-906e-0c39dbddfef9\") " Nov 02 23:58:35.567334 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:35.567306 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0751000b-140a-4c61-906e-0c39dbddfef9-plugins-config-volume" pod "0751000b-140a-4c61-906e-0c39dbddfef9" (UID: "0751000b-140a-4c61-906e-0c39dbddfef9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:58:35.567698 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:35.567675 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0751000b-140a-4c61-906e-0c39dbddfef9-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:58:35.569979 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:35.569947 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0751000b-140a-4c61-906e-0c39dbddfef9-kube-api-access-tqjq4" pod "0751000b-140a-4c61-906e-0c39dbddfef9" (UID: "0751000b-140a-4c61-906e-0c39dbddfef9"). InnerVolumeSpecName "kube-api-access-tqjq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:58:35.585941 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:35.585916 5727 generic.go:356] "Generic (PLEG): container finished" podID="830b5c2a-dd4e-4252-b9e8-c9afe2fbda88" containerID="9c53bb2c803a401b3fc016949aa5a44c584bf7e64c6fb220bdbfac7ad90afc92" exitCode=0 Nov 02 23:58:35.586065 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:35.586037 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7097dd-54699f6fdc-99kgm" event={"ID":"830b5c2a-dd4e-4252-b9e8-c9afe2fbda88","Type":"ContainerDied","Data":"9c53bb2c803a401b3fc016949aa5a44c584bf7e64c6fb220bdbfac7ad90afc92"} Nov 02 23:58:35.587961 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:35.587937 5727 generic.go:356] "Generic (PLEG): container finished" podID="0751000b-140a-4c61-906e-0c39dbddfef9" containerID="2f2058f826925cd0d9c0291cc3ca15fa114d34505d9c0705f5926353f18cd74d" exitCode=0 Nov 02 23:58:35.588057 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:35.588025 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7097dd-74ff8666cf-9vbkm" event={"ID":"0751000b-140a-4c61-906e-0c39dbddfef9","Type":"ContainerDied","Data":"2f2058f826925cd0d9c0291cc3ca15fa114d34505d9c0705f5926353f18cd74d"} Nov 02 23:58:35.588154 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:35.588065 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7097dd-74ff8666cf-9vbkm" event={"ID":"0751000b-140a-4c61-906e-0c39dbddfef9","Type":"ContainerDied","Data":"4bfa3fc6d326d2d43d77152e0658a8a3f5485a63df0c2f5b69841ea8950efcf1"} Nov 02 23:58:35.588154 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:35.588096 5727 scope.go:122] "RemoveContainer" containerID="2f2058f826925cd0d9c0291cc3ca15fa114d34505d9c0705f5926353f18cd74d" Nov 02 23:58:35.588334 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:35.588281 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7097dd-74ff8666cf-9vbkm" Nov 02 23:58:35.611166 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:35.610368 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7097dd-54699f6fdc-99kgm" Nov 02 23:58:35.611851 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:35.611829 5727 scope.go:122] "RemoveContainer" containerID="2f2058f826925cd0d9c0291cc3ca15fa114d34505d9c0705f5926353f18cd74d" Nov 02 23:58:35.612207 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:58:35.612117 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f2058f826925cd0d9c0291cc3ca15fa114d34505d9c0705f5926353f18cd74d\": container with ID starting with 2f2058f826925cd0d9c0291cc3ca15fa114d34505d9c0705f5926353f18cd74d not found: ID does not exist" containerID="2f2058f826925cd0d9c0291cc3ca15fa114d34505d9c0705f5926353f18cd74d" Nov 02 23:58:35.612309 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:35.612211 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f2058f826925cd0d9c0291cc3ca15fa114d34505d9c0705f5926353f18cd74d"} err="failed to get container status \"2f2058f826925cd0d9c0291cc3ca15fa114d34505d9c0705f5926353f18cd74d\": rpc error: code = NotFound desc = could not find container \"2f2058f826925cd0d9c0291cc3ca15fa114d34505d9c0705f5926353f18cd74d\": container with ID starting with 2f2058f826925cd0d9c0291cc3ca15fa114d34505d9c0705f5926353f18cd74d not found: ID does not exist" Nov 02 23:58:35.669067 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:35.669026 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/830b5c2a-dd4e-4252-b9e8-c9afe2fbda88-kube-api-access-b42mb\" (UniqueName: \"kubernetes.io/projected/830b5c2a-dd4e-4252-b9e8-c9afe2fbda88-kube-api-access-b42mb\") pod \"830b5c2a-dd4e-4252-b9e8-c9afe2fbda88\" (UID: \"830b5c2a-dd4e-4252-b9e8-c9afe2fbda88\") " Nov 02 23:58:35.669430 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:35.669268 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/830b5c2a-dd4e-4252-b9e8-c9afe2fbda88-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/830b5c2a-dd4e-4252-b9e8-c9afe2fbda88-plugins-config-volume\") pod \"830b5c2a-dd4e-4252-b9e8-c9afe2fbda88\" (UID: \"830b5c2a-dd4e-4252-b9e8-c9afe2fbda88\") " Nov 02 23:58:35.669756 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:35.669726 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/830b5c2a-dd4e-4252-b9e8-c9afe2fbda88-plugins-config-volume" pod "830b5c2a-dd4e-4252-b9e8-c9afe2fbda88" (UID: "830b5c2a-dd4e-4252-b9e8-c9afe2fbda88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:58:35.669881 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:35.669767 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqjq4\" (UniqueName: \"kubernetes.io/projected/0751000b-140a-4c61-906e-0c39dbddfef9-kube-api-access-tqjq4\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:58:35.672083 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:35.672051 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/830b5c2a-dd4e-4252-b9e8-c9afe2fbda88-kube-api-access-b42mb" pod "830b5c2a-dd4e-4252-b9e8-c9afe2fbda88" (UID: "830b5c2a-dd4e-4252-b9e8-c9afe2fbda88"). InnerVolumeSpecName "kube-api-access-b42mb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:58:35.731497 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:35.731434 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7097dd-74ff8666cf-9vbkm"] Nov 02 23:58:35.736580 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:35.736554 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7097dd-74ff8666cf-9vbkm"] Nov 02 23:58:35.771077 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:35.771045 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/830b5c2a-dd4e-4252-b9e8-c9afe2fbda88-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:58:35.771418 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:35.771080 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b42mb\" (UniqueName: \"kubernetes.io/projected/830b5c2a-dd4e-4252-b9e8-c9afe2fbda88-kube-api-access-b42mb\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:58:35.886781 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:35.886737 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0751000b-140a-4c61-906e-0c39dbddfef9" path="/var/lib/kubelet/pods/0751000b-140a-4c61-906e-0c39dbddfef9/volumes" Nov 02 23:58:35.887384 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:35.887362 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12cd1a89-fec7-4b82-b481-1022d5cbdd32" path="/var/lib/kubelet/pods/12cd1a89-fec7-4b82-b481-1022d5cbdd32/volumes" Nov 02 23:58:36.599247 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:36.599217 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7097dd-54699f6fdc-99kgm" Nov 02 23:58:36.599466 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:36.599217 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7097dd-54699f6fdc-99kgm" event={"ID":"830b5c2a-dd4e-4252-b9e8-c9afe2fbda88","Type":"ContainerDied","Data":"8782211ec14eb829959fbb9e9ed1d00df630c41194d9639568b647c66ce8513b"} Nov 02 23:58:36.599466 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:36.599376 5727 scope.go:122] "RemoveContainer" containerID="9c53bb2c803a401b3fc016949aa5a44c584bf7e64c6fb220bdbfac7ad90afc92" Nov 02 23:58:36.765535 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:36.765498 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7097dd-54699f6fdc-99kgm"] Nov 02 23:58:36.816198 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:36.816162 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7097dd-54699f6fdc-99kgm"] Nov 02 23:58:37.884202 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:58:37.884172 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="830b5c2a-dd4e-4252-b9e8-c9afe2fbda88" path="/var/lib/kubelet/pods/830b5c2a-dd4e-4252-b9e8-c9afe2fbda88/volumes" Nov 02 23:58:57.845750 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:58:57.845713 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:59:01.006951 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:01.006917 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h"] Nov 02 23:59:01.007370 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:01.007166 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h" podUID="660773e5-8de6-423d-9718-71742230175a" containerName="maas-api" containerID="cri-o://c79f68826ceb9b9b937d9987e6bf2b5718e46be01bc0dc3f0e0b8ed75e0cc115" gracePeriod=30 Nov 02 23:59:01.347266 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:01.347240 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h" Nov 02 23:59:01.418224 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:01.418185 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/660773e5-8de6-423d-9718-71742230175a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/660773e5-8de6-423d-9718-71742230175a-maas-api-tls\") pod \"660773e5-8de6-423d-9718-71742230175a\" (UID: \"660773e5-8de6-423d-9718-71742230175a\") " Nov 02 23:59:01.418449 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:01.418253 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660773e5-8de6-423d-9718-71742230175a-kube-api-access-sg52q\" (UniqueName: \"kubernetes.io/projected/660773e5-8de6-423d-9718-71742230175a-kube-api-access-sg52q\") pod \"660773e5-8de6-423d-9718-71742230175a\" (UID: \"660773e5-8de6-423d-9718-71742230175a\") " Nov 02 23:59:01.418758 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:01.418732 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/660773e5-8de6-423d-9718-71742230175a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/660773e5-8de6-423d-9718-71742230175a-openshift-service-ca\") pod \"660773e5-8de6-423d-9718-71742230175a\" (UID: \"660773e5-8de6-423d-9718-71742230175a\") " Nov 02 23:59:01.418867 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:01.418793 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/660773e5-8de6-423d-9718-71742230175a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/660773e5-8de6-423d-9718-71742230175a-metrics-certs\") pod \"660773e5-8de6-423d-9718-71742230175a\" (UID: \"660773e5-8de6-423d-9718-71742230175a\") " Nov 02 23:59:01.419852 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:01.419826 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660773e5-8de6-423d-9718-71742230175a-openshift-service-ca" pod "660773e5-8de6-423d-9718-71742230175a" (UID: "660773e5-8de6-423d-9718-71742230175a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:59:01.421620 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:01.421596 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/660773e5-8de6-423d-9718-71742230175a-maas-api-tls" pod "660773e5-8de6-423d-9718-71742230175a" (UID: "660773e5-8de6-423d-9718-71742230175a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:59:01.421989 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:01.421630 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660773e5-8de6-423d-9718-71742230175a-kube-api-access-sg52q" pod "660773e5-8de6-423d-9718-71742230175a" (UID: "660773e5-8de6-423d-9718-71742230175a"). InnerVolumeSpecName "kube-api-access-sg52q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:59:01.422493 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:01.422469 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/660773e5-8de6-423d-9718-71742230175a-metrics-certs" pod "660773e5-8de6-423d-9718-71742230175a" (UID: "660773e5-8de6-423d-9718-71742230175a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:59:01.521365 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:01.521322 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/660773e5-8de6-423d-9718-71742230175a-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:01.521365 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:01.521359 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/660773e5-8de6-423d-9718-71742230175a-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:01.521726 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:01.521373 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/660773e5-8de6-423d-9718-71742230175a-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:01.521726 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:01.521387 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg52q\" (UniqueName: \"kubernetes.io/projected/660773e5-8de6-423d-9718-71742230175a-kube-api-access-sg52q\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:01.806033 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:01.805997 5727 generic.go:356] "Generic (PLEG): container finished" podID="660773e5-8de6-423d-9718-71742230175a" containerID="c79f68826ceb9b9b937d9987e6bf2b5718e46be01bc0dc3f0e0b8ed75e0cc115" exitCode=0 Nov 02 23:59:01.806234 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:01.806060 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h" Nov 02 23:59:01.806234 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:01.806087 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h" event={"ID":"660773e5-8de6-423d-9718-71742230175a","Type":"ContainerDied","Data":"c79f68826ceb9b9b937d9987e6bf2b5718e46be01bc0dc3f0e0b8ed75e0cc115"} Nov 02 23:59:01.806234 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:01.806157 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h" event={"ID":"660773e5-8de6-423d-9718-71742230175a","Type":"ContainerDied","Data":"fb70efad19e0563572f81e81bca7f8b2e8c9edab250bf1f6db66fcf9862b9ea2"} Nov 02 23:59:01.806234 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:01.806184 5727 scope.go:122] "RemoveContainer" containerID="c79f68826ceb9b9b937d9987e6bf2b5718e46be01bc0dc3f0e0b8ed75e0cc115" Nov 02 23:59:01.823978 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:01.823956 5727 scope.go:122] "RemoveContainer" containerID="c79f68826ceb9b9b937d9987e6bf2b5718e46be01bc0dc3f0e0b8ed75e0cc115" Nov 02 23:59:01.824343 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:59:01.824312 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c79f68826ceb9b9b937d9987e6bf2b5718e46be01bc0dc3f0e0b8ed75e0cc115\": container with ID starting with c79f68826ceb9b9b937d9987e6bf2b5718e46be01bc0dc3f0e0b8ed75e0cc115 not found: ID does not exist" containerID="c79f68826ceb9b9b937d9987e6bf2b5718e46be01bc0dc3f0e0b8ed75e0cc115" Nov 02 23:59:01.824434 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:01.824344 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c79f68826ceb9b9b937d9987e6bf2b5718e46be01bc0dc3f0e0b8ed75e0cc115"} err="failed to get container status \"c79f68826ceb9b9b937d9987e6bf2b5718e46be01bc0dc3f0e0b8ed75e0cc115\": rpc error: code = NotFound desc = could not find container \"c79f68826ceb9b9b937d9987e6bf2b5718e46be01bc0dc3f0e0b8ed75e0cc115\": container with ID starting with c79f68826ceb9b9b937d9987e6bf2b5718e46be01bc0dc3f0e0b8ed75e0cc115 not found: ID does not exist" Nov 02 23:59:01.989252 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:01.989215 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h"] Nov 02 23:59:02.036364 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:02.036323 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-18a6d8-787df88cc8-nzf9h"] Nov 02 23:59:03.923223 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:03.915697 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="660773e5-8de6-423d-9718-71742230175a" path="/var/lib/kubelet/pods/660773e5-8de6-423d-9718-71742230175a/volumes" Nov 02 23:59:27.846266 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:59:27.846228 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 02 23:59:27.953153 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:27.953096 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8"] Nov 02 23:59:27.953310 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:27.953293 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8" podUID="097f0270-0698-4864-b784-b702e76a7159" containerName="maas-api" containerID="cri-o://0746dab54d71f24875fdf5bad6d646edac98ca7cb3e4f783393f6060053dccef" gracePeriod=30 Nov 02 23:59:28.244237 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:28.244209 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8" Nov 02 23:59:28.306478 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:28.306438 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/097f0270-0698-4864-b784-b702e76a7159-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/097f0270-0698-4864-b784-b702e76a7159-maas-api-tls\") pod \"097f0270-0698-4864-b784-b702e76a7159\" (UID: \"097f0270-0698-4864-b784-b702e76a7159\") " Nov 02 23:59:28.306645 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:28.306560 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/097f0270-0698-4864-b784-b702e76a7159-metrics-certs\" (UniqueName: \"kubernetes.io/secret/097f0270-0698-4864-b784-b702e76a7159-metrics-certs\") pod \"097f0270-0698-4864-b784-b702e76a7159\" (UID: \"097f0270-0698-4864-b784-b702e76a7159\") " Nov 02 23:59:28.306645 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:28.306597 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/097f0270-0698-4864-b784-b702e76a7159-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/097f0270-0698-4864-b784-b702e76a7159-openshift-service-ca\") pod \"097f0270-0698-4864-b784-b702e76a7159\" (UID: \"097f0270-0698-4864-b784-b702e76a7159\") " Nov 02 23:59:28.306912 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:28.306836 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/097f0270-0698-4864-b784-b702e76a7159-kube-api-access-tg5wf\" (UniqueName: \"kubernetes.io/projected/097f0270-0698-4864-b784-b702e76a7159-kube-api-access-tg5wf\") pod \"097f0270-0698-4864-b784-b702e76a7159\" (UID: \"097f0270-0698-4864-b784-b702e76a7159\") " Nov 02 23:59:28.307061 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:28.307036 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/097f0270-0698-4864-b784-b702e76a7159-openshift-service-ca" pod "097f0270-0698-4864-b784-b702e76a7159" (UID: "097f0270-0698-4864-b784-b702e76a7159"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:59:28.307413 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:28.307392 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/097f0270-0698-4864-b784-b702e76a7159-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:28.309332 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:28.309307 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/097f0270-0698-4864-b784-b702e76a7159-kube-api-access-tg5wf" pod "097f0270-0698-4864-b784-b702e76a7159" (UID: "097f0270-0698-4864-b784-b702e76a7159"). InnerVolumeSpecName "kube-api-access-tg5wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:59:28.309457 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:28.309431 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/097f0270-0698-4864-b784-b702e76a7159-metrics-certs" pod "097f0270-0698-4864-b784-b702e76a7159" (UID: "097f0270-0698-4864-b784-b702e76a7159"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:59:28.309553 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:28.309489 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/097f0270-0698-4864-b784-b702e76a7159-maas-api-tls" pod "097f0270-0698-4864-b784-b702e76a7159" (UID: "097f0270-0698-4864-b784-b702e76a7159"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:59:28.408943 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:28.408873 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg5wf\" (UniqueName: \"kubernetes.io/projected/097f0270-0698-4864-b784-b702e76a7159-kube-api-access-tg5wf\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:28.408943 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:28.408902 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/097f0270-0698-4864-b784-b702e76a7159-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:28.408943 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:28.408912 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/097f0270-0698-4864-b784-b702e76a7159-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:29.044305 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:29.044263 5727 generic.go:356] "Generic (PLEG): container finished" podID="097f0270-0698-4864-b784-b702e76a7159" containerID="0746dab54d71f24875fdf5bad6d646edac98ca7cb3e4f783393f6060053dccef" exitCode=0 Nov 02 23:59:29.044736 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:29.044315 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8" Nov 02 23:59:29.044736 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:29.044333 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8" event={"ID":"097f0270-0698-4864-b784-b702e76a7159","Type":"ContainerDied","Data":"0746dab54d71f24875fdf5bad6d646edac98ca7cb3e4f783393f6060053dccef"} Nov 02 23:59:29.044736 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:29.044400 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8" event={"ID":"097f0270-0698-4864-b784-b702e76a7159","Type":"ContainerDied","Data":"eb4e6bd93bd8e13b73d7b237b4665fe2088c24008987e39ec38bd0866c2536bf"} Nov 02 23:59:29.044736 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:29.044425 5727 scope.go:122] "RemoveContainer" containerID="0746dab54d71f24875fdf5bad6d646edac98ca7cb3e4f783393f6060053dccef" Nov 02 23:59:29.057692 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:29.057669 5727 scope.go:122] "RemoveContainer" containerID="0746dab54d71f24875fdf5bad6d646edac98ca7cb3e4f783393f6060053dccef" Nov 02 23:59:29.057977 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:59:29.057928 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0746dab54d71f24875fdf5bad6d646edac98ca7cb3e4f783393f6060053dccef\": container with ID starting with 0746dab54d71f24875fdf5bad6d646edac98ca7cb3e4f783393f6060053dccef not found: ID does not exist" containerID="0746dab54d71f24875fdf5bad6d646edac98ca7cb3e4f783393f6060053dccef" Nov 02 23:59:29.058055 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:29.057985 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0746dab54d71f24875fdf5bad6d646edac98ca7cb3e4f783393f6060053dccef"} err="failed to get container status \"0746dab54d71f24875fdf5bad6d646edac98ca7cb3e4f783393f6060053dccef\": rpc error: code = NotFound desc = could not find container \"0746dab54d71f24875fdf5bad6d646edac98ca7cb3e4f783393f6060053dccef\": container with ID starting with 0746dab54d71f24875fdf5bad6d646edac98ca7cb3e4f783393f6060053dccef not found: ID does not exist" Nov 02 23:59:29.210713 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:29.210682 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8"] Nov 02 23:59:29.214490 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:29.214466 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f7002-fc6fb9f9-qknk8"] Nov 02 23:59:29.895031 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:29.894986 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="097f0270-0698-4864-b784-b702e76a7159" path="/var/lib/kubelet/pods/097f0270-0698-4864-b784-b702e76a7159/volumes" Nov 02 23:59:45.282153 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:45.274939 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5e7e9e-6cc6b59469-q88rm"] Nov 02 23:59:45.401156 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:45.398350 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5e7e9e-66b9694f59-5rtsh"] Nov 02 23:59:50.282611 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:50.282562 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-5e7e9e-6cc6b59469-q88rm" podUID="05c07885-2e05-4f96-b72d-2e2a3d918bc5" containerName="payload-processing" containerID="cri-o://355ab65ce59c6d1f761e12aca4384e5b5aa740180f5ab69c5e873e952985e91d" gracePeriod=25 Nov 02 23:59:50.398900 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:50.398833 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5e7e9e-66b9694f59-5rtsh" podUID="432f3e59-fed9-4cc8-9e48-403f02f0c40e" containerName="payload-pre-processing" containerID="cri-o://b566d3df62728bcac0bab3091d6fed53d64da29aa5203a04d5101e626cc4e627" gracePeriod=25 Nov 02 23:59:50.609630 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:50.609237 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-5e7e9e-6cc6b59469-q88rm" Nov 02 23:59:50.674185 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:50.674149 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05c07885-2e05-4f96-b72d-2e2a3d918bc5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c07885-2e05-4f96-b72d-2e2a3d918bc5-plugins-config-volume\") pod \"05c07885-2e05-4f96-b72d-2e2a3d918bc5\" (UID: \"05c07885-2e05-4f96-b72d-2e2a3d918bc5\") " Nov 02 23:59:50.674391 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:50.674331 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05c07885-2e05-4f96-b72d-2e2a3d918bc5-kube-api-access-7wmch\" (UniqueName: \"kubernetes.io/projected/05c07885-2e05-4f96-b72d-2e2a3d918bc5-kube-api-access-7wmch\") pod \"05c07885-2e05-4f96-b72d-2e2a3d918bc5\" (UID: \"05c07885-2e05-4f96-b72d-2e2a3d918bc5\") " Nov 02 23:59:50.674657 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:50.674626 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05c07885-2e05-4f96-b72d-2e2a3d918bc5-plugins-config-volume" pod "05c07885-2e05-4f96-b72d-2e2a3d918bc5" (UID: "05c07885-2e05-4f96-b72d-2e2a3d918bc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:59:50.674950 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:50.674926 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c07885-2e05-4f96-b72d-2e2a3d918bc5-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:50.677518 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:50.677488 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c07885-2e05-4f96-b72d-2e2a3d918bc5-kube-api-access-7wmch" pod "05c07885-2e05-4f96-b72d-2e2a3d918bc5" (UID: "05c07885-2e05-4f96-b72d-2e2a3d918bc5"). InnerVolumeSpecName "kube-api-access-7wmch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:59:50.698292 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:50.698258 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5e7e9e-66b9694f59-5rtsh" Nov 02 23:59:50.775856 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:50.775814 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/432f3e59-fed9-4cc8-9e48-403f02f0c40e-kube-api-access-f2tqh\" (UniqueName: \"kubernetes.io/projected/432f3e59-fed9-4cc8-9e48-403f02f0c40e-kube-api-access-f2tqh\") pod \"432f3e59-fed9-4cc8-9e48-403f02f0c40e\" (UID: \"432f3e59-fed9-4cc8-9e48-403f02f0c40e\") " Nov 02 23:59:50.776061 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:50.775865 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/432f3e59-fed9-4cc8-9e48-403f02f0c40e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432f3e59-fed9-4cc8-9e48-403f02f0c40e-plugins-config-volume\") pod \"432f3e59-fed9-4cc8-9e48-403f02f0c40e\" (UID: \"432f3e59-fed9-4cc8-9e48-403f02f0c40e\") " Nov 02 23:59:50.776277 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:50.776258 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wmch\" (UniqueName: \"kubernetes.io/projected/05c07885-2e05-4f96-b72d-2e2a3d918bc5-kube-api-access-7wmch\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:50.776373 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:50.776319 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/432f3e59-fed9-4cc8-9e48-403f02f0c40e-plugins-config-volume" pod "432f3e59-fed9-4cc8-9e48-403f02f0c40e" (UID: "432f3e59-fed9-4cc8-9e48-403f02f0c40e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:59:50.778298 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:50.778263 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432f3e59-fed9-4cc8-9e48-403f02f0c40e-kube-api-access-f2tqh" pod "432f3e59-fed9-4cc8-9e48-403f02f0c40e" (UID: "432f3e59-fed9-4cc8-9e48-403f02f0c40e"). InnerVolumeSpecName "kube-api-access-f2tqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:59:50.877711 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:50.877615 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2tqh\" (UniqueName: \"kubernetes.io/projected/432f3e59-fed9-4cc8-9e48-403f02f0c40e-kube-api-access-f2tqh\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:50.877711 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:50.877656 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432f3e59-fed9-4cc8-9e48-403f02f0c40e-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:51.240597 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:51.240502 5727 generic.go:356] "Generic (PLEG): container finished" podID="05c07885-2e05-4f96-b72d-2e2a3d918bc5" containerID="355ab65ce59c6d1f761e12aca4384e5b5aa740180f5ab69c5e873e952985e91d" exitCode=0 Nov 02 23:59:51.240597 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:51.240553 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-5e7e9e-6cc6b59469-q88rm" Nov 02 23:59:51.240800 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:51.240587 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-5e7e9e-6cc6b59469-q88rm" event={"ID":"05c07885-2e05-4f96-b72d-2e2a3d918bc5","Type":"ContainerDied","Data":"355ab65ce59c6d1f761e12aca4384e5b5aa740180f5ab69c5e873e952985e91d"} Nov 02 23:59:51.240800 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:51.240647 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-5e7e9e-6cc6b59469-q88rm" event={"ID":"05c07885-2e05-4f96-b72d-2e2a3d918bc5","Type":"ContainerDied","Data":"1b656ea22fb907b7cbd97e5f0509daa265caee6630c33bb8fe36726f0333cb32"} Nov 02 23:59:51.240800 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:51.240670 5727 scope.go:122] "RemoveContainer" containerID="355ab65ce59c6d1f761e12aca4384e5b5aa740180f5ab69c5e873e952985e91d" Nov 02 23:59:51.242354 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:51.242321 5727 generic.go:356] "Generic (PLEG): container finished" podID="432f3e59-fed9-4cc8-9e48-403f02f0c40e" containerID="b566d3df62728bcac0bab3091d6fed53d64da29aa5203a04d5101e626cc4e627" exitCode=0 Nov 02 23:59:51.242470 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:51.242386 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5e7e9e-66b9694f59-5rtsh" Nov 02 23:59:51.242535 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:51.242414 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5e7e9e-66b9694f59-5rtsh" event={"ID":"432f3e59-fed9-4cc8-9e48-403f02f0c40e","Type":"ContainerDied","Data":"b566d3df62728bcac0bab3091d6fed53d64da29aa5203a04d5101e626cc4e627"} Nov 02 23:59:51.242580 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:51.242566 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5e7e9e-66b9694f59-5rtsh" event={"ID":"432f3e59-fed9-4cc8-9e48-403f02f0c40e","Type":"ContainerDied","Data":"f1a92b68ad656519db77fdd11167e19dd958eebd1191347c2be899c3bea5570b"} Nov 02 23:59:51.251970 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:51.251952 5727 scope.go:122] "RemoveContainer" containerID="355ab65ce59c6d1f761e12aca4384e5b5aa740180f5ab69c5e873e952985e91d" Nov 02 23:59:51.252337 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:59:51.252318 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"355ab65ce59c6d1f761e12aca4384e5b5aa740180f5ab69c5e873e952985e91d\": container with ID starting with 355ab65ce59c6d1f761e12aca4384e5b5aa740180f5ab69c5e873e952985e91d not found: ID does not exist" containerID="355ab65ce59c6d1f761e12aca4384e5b5aa740180f5ab69c5e873e952985e91d" Nov 02 23:59:51.252413 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:51.252348 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"355ab65ce59c6d1f761e12aca4384e5b5aa740180f5ab69c5e873e952985e91d"} err="failed to get container status \"355ab65ce59c6d1f761e12aca4384e5b5aa740180f5ab69c5e873e952985e91d\": rpc error: code = NotFound desc = could not find container \"355ab65ce59c6d1f761e12aca4384e5b5aa740180f5ab69c5e873e952985e91d\": container with ID starting with 355ab65ce59c6d1f761e12aca4384e5b5aa740180f5ab69c5e873e952985e91d not found: ID does not exist" Nov 02 23:59:51.252413 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:51.252367 5727 scope.go:122] "RemoveContainer" containerID="b566d3df62728bcac0bab3091d6fed53d64da29aa5203a04d5101e626cc4e627" Nov 02 23:59:51.270324 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:51.270299 5727 scope.go:122] "RemoveContainer" containerID="b566d3df62728bcac0bab3091d6fed53d64da29aa5203a04d5101e626cc4e627" Nov 02 23:59:51.270699 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:59:51.270677 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b566d3df62728bcac0bab3091d6fed53d64da29aa5203a04d5101e626cc4e627\": container with ID starting with b566d3df62728bcac0bab3091d6fed53d64da29aa5203a04d5101e626cc4e627 not found: ID does not exist" containerID="b566d3df62728bcac0bab3091d6fed53d64da29aa5203a04d5101e626cc4e627" Nov 02 23:59:51.270795 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:51.270712 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b566d3df62728bcac0bab3091d6fed53d64da29aa5203a04d5101e626cc4e627"} err="failed to get container status \"b566d3df62728bcac0bab3091d6fed53d64da29aa5203a04d5101e626cc4e627\": rpc error: code = NotFound desc = could not find container \"b566d3df62728bcac0bab3091d6fed53d64da29aa5203a04d5101e626cc4e627\": container with ID starting with b566d3df62728bcac0bab3091d6fed53d64da29aa5203a04d5101e626cc4e627 not found: ID does not exist" Nov 02 23:59:51.407265 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:51.407228 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5e7e9e-66b9694f59-5rtsh"] Nov 02 23:59:51.456680 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:51.456643 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5e7e9e-66b9694f59-5rtsh"] Nov 02 23:59:51.518080 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:51.517996 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5e7e9e-6cc6b59469-q88rm"] Nov 02 23:59:51.524264 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:51.524233 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5e7e9e-6cc6b59469-q88rm"] Nov 02 23:59:51.886324 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:51.886293 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05c07885-2e05-4f96-b72d-2e2a3d918bc5" path="/var/lib/kubelet/pods/05c07885-2e05-4f96-b72d-2e2a3d918bc5/volumes" Nov 02 23:59:51.886773 ip-10-0-135-176 kubenswrapper[5727]: I1102 23:59:51.886758 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="432f3e59-fed9-4cc8-9e48-403f02f0c40e" path="/var/lib/kubelet/pods/432f3e59-fed9-4cc8-9e48-403f02f0c40e/volumes" Nov 02 23:59:57.847165 ip-10-0-135-176 kubenswrapper[5727]: E1102 23:59:57.847100 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:00:00.183251 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.183201 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-pruner-29894400-htxcw"] Nov 03 00:00:00.184174 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.184151 5727 cpu_manager.go:414] "containerMap: removing container" podUID="12cd1a89-fec7-4b82-b481-1022d5cbdd32" containerName="maas-api" Nov 03 00:00:00.184279 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.184176 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12cd1a89-fec7-4b82-b481-1022d5cbdd32" containerName="maas-api" Nov 03 00:00:00.184279 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.184198 5727 cpu_manager.go:414] "containerMap: removing container" podUID="e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0" containerName="payload-processing" Nov 03 00:00:00.184279 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.184206 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0" containerName="payload-processing" Nov 03 00:00:00.184279 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.184220 5727 cpu_manager.go:414] "containerMap: removing container" podUID="830b5c2a-dd4e-4252-b9e8-c9afe2fbda88" containerName="payload-pre-processing" Nov 03 00:00:00.184279 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.184234 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="830b5c2a-dd4e-4252-b9e8-c9afe2fbda88" containerName="payload-pre-processing" Nov 03 00:00:00.184279 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.184247 5727 cpu_manager.go:414] "containerMap: removing container" podUID="097f0270-0698-4864-b784-b702e76a7159" containerName="maas-api" Nov 03 00:00:00.184279 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.184254 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="097f0270-0698-4864-b784-b702e76a7159" containerName="maas-api" Nov 03 00:00:00.184279 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.184273 5727 cpu_manager.go:414] "containerMap: removing container" podUID="aa393900-e35d-4751-817c-c6ed6a2d6a8a" containerName="payload-pre-processing" Nov 03 00:00:00.184279 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.184280 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa393900-e35d-4751-817c-c6ed6a2d6a8a" containerName="payload-pre-processing" Nov 03 00:00:00.184564 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.184305 5727 cpu_manager.go:414] "containerMap: removing container" podUID="05c07885-2e05-4f96-b72d-2e2a3d918bc5" containerName="payload-processing" Nov 03 00:00:00.184564 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.184312 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05c07885-2e05-4f96-b72d-2e2a3d918bc5" containerName="payload-processing" Nov 03 00:00:00.184564 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.184322 5727 cpu_manager.go:414] "containerMap: removing container" podUID="432f3e59-fed9-4cc8-9e48-403f02f0c40e" containerName="payload-pre-processing" Nov 03 00:00:00.184564 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.184329 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="432f3e59-fed9-4cc8-9e48-403f02f0c40e" containerName="payload-pre-processing" Nov 03 00:00:00.184564 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.184345 5727 cpu_manager.go:414] "containerMap: removing container" podUID="0751000b-140a-4c61-906e-0c39dbddfef9" containerName="payload-processing" Nov 03 00:00:00.184564 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.184351 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0751000b-140a-4c61-906e-0c39dbddfef9" containerName="payload-processing" Nov 03 00:00:00.184564 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.184381 5727 cpu_manager.go:414] "containerMap: removing container" podUID="78557250-4ce3-404d-b057-e66bde4fec4d" containerName="payload-processing" Nov 03 00:00:00.184564 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.184387 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78557250-4ce3-404d-b057-e66bde4fec4d" containerName="payload-processing" Nov 03 00:00:00.184564 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.184403 5727 cpu_manager.go:414] "containerMap: removing container" podUID="660773e5-8de6-423d-9718-71742230175a" containerName="maas-api" Nov 03 00:00:00.184564 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.184409 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660773e5-8de6-423d-9718-71742230175a" containerName="maas-api" Nov 03 00:00:00.184860 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.184576 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="660773e5-8de6-423d-9718-71742230175a" containerName="maas-api" Nov 03 00:00:00.184860 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.184590 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="12cd1a89-fec7-4b82-b481-1022d5cbdd32" containerName="maas-api" Nov 03 00:00:00.184860 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.184599 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="e84b19b9-9f74-4d5d-ab9f-dbbefd6d06f0" containerName="payload-processing" Nov 03 00:00:00.184860 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.184607 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="830b5c2a-dd4e-4252-b9e8-c9afe2fbda88" containerName="payload-pre-processing" Nov 03 00:00:00.184860 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.184614 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="097f0270-0698-4864-b784-b702e76a7159" containerName="maas-api" Nov 03 00:00:00.184860 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.184621 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa393900-e35d-4751-817c-c6ed6a2d6a8a" containerName="payload-pre-processing" Nov 03 00:00:00.184860 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.184628 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="05c07885-2e05-4f96-b72d-2e2a3d918bc5" containerName="payload-processing" Nov 03 00:00:00.184860 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.184634 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="432f3e59-fed9-4cc8-9e48-403f02f0c40e" containerName="payload-pre-processing" Nov 03 00:00:00.184860 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.184640 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="0751000b-140a-4c61-906e-0c39dbddfef9" containerName="payload-processing" Nov 03 00:00:00.184860 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.184647 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="78557250-4ce3-404d-b057-e66bde4fec4d" containerName="payload-processing" Nov 03 00:00:00.189607 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.189587 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29894400-htxcw" Nov 03 00:00:00.215371 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.215343 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962"] Nov 03 00:00:00.220327 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.220304 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962" Nov 03 00:00:00.234909 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.234880 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9"] Nov 03 00:00:00.239806 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.239782 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9" Nov 03 00:00:00.241492 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.241466 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7"] Nov 03 00:00:00.246303 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.246281 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7" Nov 03 00:00:00.249231 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.249098 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt"] Nov 03 00:00:00.253424 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.253405 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt" Nov 03 00:00:00.261243 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.261107 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"serviceca\"" Nov 03 00:00:00.285272 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.285245 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29894400-htxcw"] Nov 03 00:00:00.285566 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.285544 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ae3e4d01-001e-4493-b7ac-7cab5bde77a0-serviceca\") pod \"image-pruner-29894400-htxcw\" (UID: \"ae3e4d01-001e-4493-b7ac-7cab5bde77a0\") " pod="openshift-image-registry/image-pruner-29894400-htxcw" Nov 03 00:00:00.285647 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.285603 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qccdb\" (UniqueName: \"kubernetes.io/projected/e56f2e10-bb0e-474a-9611-7ef9d6d22cc7-kube-api-access-qccdb\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962\" (UID: \"e56f2e10-bb0e-474a-9611-7ef9d6d22cc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962" Nov 03 00:00:00.285777 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.285759 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc2n9\" (UniqueName: \"kubernetes.io/projected/ae3e4d01-001e-4493-b7ac-7cab5bde77a0-kube-api-access-nc2n9\") pod \"image-pruner-29894400-htxcw\" (UID: \"ae3e4d01-001e-4493-b7ac-7cab5bde77a0\") " pod="openshift-image-registry/image-pruner-29894400-htxcw" Nov 03 00:00:00.285959 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.285940 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e56f2e10-bb0e-474a-9611-7ef9d6d22cc7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962\" (UID: \"e56f2e10-bb0e-474a-9611-7ef9d6d22cc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962" Nov 03 00:00:00.339531 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.339496 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"pruner-dockercfg-vrt7s\"" Nov 03 00:00:00.339729 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.339541 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 03 00:00:00.389648 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.387983 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e56f2e10-bb0e-474a-9611-7ef9d6d22cc7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962\" (UID: \"e56f2e10-bb0e-474a-9611-7ef9d6d22cc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962" Nov 03 00:00:00.389648 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.388116 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ae3e4d01-001e-4493-b7ac-7cab5bde77a0-serviceca\") pod \"image-pruner-29894400-htxcw\" (UID: \"ae3e4d01-001e-4493-b7ac-7cab5bde77a0\") " pod="openshift-image-registry/image-pruner-29894400-htxcw" Nov 03 00:00:00.389648 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.388190 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qccdb\" (UniqueName: \"kubernetes.io/projected/e56f2e10-bb0e-474a-9611-7ef9d6d22cc7-kube-api-access-qccdb\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962\" (UID: \"e56f2e10-bb0e-474a-9611-7ef9d6d22cc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962" Nov 03 00:00:00.389648 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.388225 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbl4d\" (UniqueName: \"kubernetes.io/projected/bf91e63c-e756-4218-8de0-8a22a4664b24-kube-api-access-jbl4d\") pod \"maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt\" (UID: \"bf91e63c-e756-4218-8de0-8a22a4664b24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt" Nov 03 00:00:00.389648 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.388299 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf91e63c-e756-4218-8de0-8a22a4664b24-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt\" (UID: \"bf91e63c-e756-4218-8de0-8a22a4664b24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt" Nov 03 00:00:00.389648 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.388334 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz8nc\" (UniqueName: \"kubernetes.io/projected/9fad0b49-d23b-4913-b0fa-87e1c55c5a20-kube-api-access-xz8nc\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9\" (UID: \"9fad0b49-d23b-4913-b0fa-87e1c55c5a20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9" Nov 03 00:00:00.389648 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.388374 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc2n9\" (UniqueName: \"kubernetes.io/projected/ae3e4d01-001e-4493-b7ac-7cab5bde77a0-kube-api-access-nc2n9\") pod \"image-pruner-29894400-htxcw\" (UID: \"ae3e4d01-001e-4493-b7ac-7cab5bde77a0\") " pod="openshift-image-registry/image-pruner-29894400-htxcw" Nov 03 00:00:00.389648 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.388403 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvr4l\" (UniqueName: \"kubernetes.io/projected/c10cf5b1-d301-48d1-89a5-cac200b396d7-kube-api-access-hvr4l\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7\" (UID: \"c10cf5b1-d301-48d1-89a5-cac200b396d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7" Nov 03 00:00:00.389648 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.388470 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c10cf5b1-d301-48d1-89a5-cac200b396d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7\" (UID: \"c10cf5b1-d301-48d1-89a5-cac200b396d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7" Nov 03 00:00:00.389648 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.388850 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e56f2e10-bb0e-474a-9611-7ef9d6d22cc7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962\" (UID: \"e56f2e10-bb0e-474a-9611-7ef9d6d22cc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962" Nov 03 00:00:00.389648 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.389181 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fad0b49-d23b-4913-b0fa-87e1c55c5a20-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9\" (UID: \"9fad0b49-d23b-4913-b0fa-87e1c55c5a20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9" Nov 03 00:00:00.389648 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.389586 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ae3e4d01-001e-4493-b7ac-7cab5bde77a0-serviceca\") pod \"image-pruner-29894400-htxcw\" (UID: \"ae3e4d01-001e-4493-b7ac-7cab5bde77a0\") " pod="openshift-image-registry/image-pruner-29894400-htxcw" Nov 03 00:00:00.391311 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.390501 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962"] Nov 03 00:00:00.396117 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.396090 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7"] Nov 03 00:00:00.449390 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.449316 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qccdb\" (UniqueName: \"kubernetes.io/projected/e56f2e10-bb0e-474a-9611-7ef9d6d22cc7-kube-api-access-qccdb\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962\" (UID: \"e56f2e10-bb0e-474a-9611-7ef9d6d22cc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962" Nov 03 00:00:00.449567 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.449421 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc2n9\" (UniqueName: \"kubernetes.io/projected/ae3e4d01-001e-4493-b7ac-7cab5bde77a0-kube-api-access-nc2n9\") pod \"image-pruner-29894400-htxcw\" (UID: \"ae3e4d01-001e-4493-b7ac-7cab5bde77a0\") " pod="openshift-image-registry/image-pruner-29894400-htxcw" Nov 03 00:00:00.490728 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.490692 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbl4d\" (UniqueName: \"kubernetes.io/projected/bf91e63c-e756-4218-8de0-8a22a4664b24-kube-api-access-jbl4d\") pod \"maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt\" (UID: \"bf91e63c-e756-4218-8de0-8a22a4664b24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt" Nov 03 00:00:00.490911 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.490765 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf91e63c-e756-4218-8de0-8a22a4664b24-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt\" (UID: \"bf91e63c-e756-4218-8de0-8a22a4664b24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt" Nov 03 00:00:00.490911 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.490800 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz8nc\" (UniqueName: \"kubernetes.io/projected/9fad0b49-d23b-4913-b0fa-87e1c55c5a20-kube-api-access-xz8nc\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9\" (UID: \"9fad0b49-d23b-4913-b0fa-87e1c55c5a20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9" Nov 03 00:00:00.490911 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.490839 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvr4l\" (UniqueName: \"kubernetes.io/projected/c10cf5b1-d301-48d1-89a5-cac200b396d7-kube-api-access-hvr4l\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7\" (UID: \"c10cf5b1-d301-48d1-89a5-cac200b396d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7" Nov 03 00:00:00.490911 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.490891 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c10cf5b1-d301-48d1-89a5-cac200b396d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7\" (UID: \"c10cf5b1-d301-48d1-89a5-cac200b396d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7" Nov 03 00:00:00.491159 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.490936 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fad0b49-d23b-4913-b0fa-87e1c55c5a20-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9\" (UID: \"9fad0b49-d23b-4913-b0fa-87e1c55c5a20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9" Nov 03 00:00:00.491677 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.491652 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fad0b49-d23b-4913-b0fa-87e1c55c5a20-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9\" (UID: \"9fad0b49-d23b-4913-b0fa-87e1c55c5a20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9" Nov 03 00:00:00.491894 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.491870 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c10cf5b1-d301-48d1-89a5-cac200b396d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7\" (UID: \"c10cf5b1-d301-48d1-89a5-cac200b396d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7" Nov 03 00:00:00.492567 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.492548 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf91e63c-e756-4218-8de0-8a22a4664b24-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt\" (UID: \"bf91e63c-e756-4218-8de0-8a22a4664b24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt" Nov 03 00:00:00.536172 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.536122 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29894400-htxcw" Nov 03 00:00:00.544610 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.544583 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962" Nov 03 00:00:00.552347 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.552318 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9"] Nov 03 00:00:00.555365 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.555340 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt"] Nov 03 00:00:00.556597 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.556567 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz8nc\" (UniqueName: \"kubernetes.io/projected/9fad0b49-d23b-4913-b0fa-87e1c55c5a20-kube-api-access-xz8nc\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9\" (UID: \"9fad0b49-d23b-4913-b0fa-87e1c55c5a20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9" Nov 03 00:00:00.557648 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.557217 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvr4l\" (UniqueName: \"kubernetes.io/projected/c10cf5b1-d301-48d1-89a5-cac200b396d7-kube-api-access-hvr4l\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7\" (UID: \"c10cf5b1-d301-48d1-89a5-cac200b396d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7" Nov 03 00:00:00.558840 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.558816 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9" Nov 03 00:00:00.567407 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.567384 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbl4d\" (UniqueName: \"kubernetes.io/projected/bf91e63c-e756-4218-8de0-8a22a4664b24-kube-api-access-jbl4d\") pod \"maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt\" (UID: \"bf91e63c-e756-4218-8de0-8a22a4664b24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt" Nov 03 00:00:00.571279 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.571253 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7" Nov 03 00:00:00.580618 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:00.580592 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt" Nov 03 00:00:01.067631 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:00:01.067600 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3e4d01_001e_4493_b7ac_7cab5bde77a0.slice/crio-3d40a726df061fd8161a929142cba38e9c95f58f909886ea86a4dc1d3f391c3d WatchSource:0}: Error finding container 3d40a726df061fd8161a929142cba38e9c95f58f909886ea86a4dc1d3f391c3d: Status 404 returned error can't find the container with id 3d40a726df061fd8161a929142cba38e9c95f58f909886ea86a4dc1d3f391c3d Nov 03 00:00:01.067751 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:01.067686 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29894400-htxcw"] Nov 03 00:00:01.077227 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:01.077200 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9"] Nov 03 00:00:01.078163 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:01.078106 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962"] Nov 03 00:00:01.115270 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:00:01.115244 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode56f2e10_bb0e_474a_9611_7ef9d6d22cc7.slice/crio-4d6abca5f5eaba65752da032e9fed7752eac98ab1e81be920107ae5aba721f12 WatchSource:0}: Error finding container 4d6abca5f5eaba65752da032e9fed7752eac98ab1e81be920107ae5aba721f12: Status 404 returned error can't find the container with id 4d6abca5f5eaba65752da032e9fed7752eac98ab1e81be920107ae5aba721f12 Nov 03 00:00:01.116979 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:00:01.116954 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fad0b49_d23b_4913_b0fa_87e1c55c5a20.slice/crio-85370a7589739de4bb76854bdf88e5425ba924bd247c7a9b91cacbb52aef0c85 WatchSource:0}: Error finding container 85370a7589739de4bb76854bdf88e5425ba924bd247c7a9b91cacbb52aef0c85: Status 404 returned error can't find the container with id 85370a7589739de4bb76854bdf88e5425ba924bd247c7a9b91cacbb52aef0c85 Nov 03 00:00:01.127407 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:01.127381 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7"] Nov 03 00:00:01.130995 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:00:01.130969 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc10cf5b1_d301_48d1_89a5_cac200b396d7.slice/crio-9ea770bdf567c97b5545a0bd405ea8cd56cbeffbdc12b7717afbbfefca7475c1 WatchSource:0}: Error finding container 9ea770bdf567c97b5545a0bd405ea8cd56cbeffbdc12b7717afbbfefca7475c1: Status 404 returned error can't find the container with id 9ea770bdf567c97b5545a0bd405ea8cd56cbeffbdc12b7717afbbfefca7475c1 Nov 03 00:00:01.131635 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:01.131607 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt"] Nov 03 00:00:01.133031 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:00:01.133009 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf91e63c_e756_4218_8de0_8a22a4664b24.slice/crio-ad478dfd63c713b74863c6cc72358c7aa90f0baef68681b9bd1da87600131698 WatchSource:0}: Error finding container ad478dfd63c713b74863c6cc72358c7aa90f0baef68681b9bd1da87600131698: Status 404 returned error can't find the container with id ad478dfd63c713b74863c6cc72358c7aa90f0baef68681b9bd1da87600131698 Nov 03 00:00:01.319075 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:01.318980 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7" event={"ID":"c10cf5b1-d301-48d1-89a5-cac200b396d7","Type":"ContainerStarted","Data":"9ea770bdf567c97b5545a0bd405ea8cd56cbeffbdc12b7717afbbfefca7475c1"} Nov 03 00:00:01.320438 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:01.320408 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt" event={"ID":"bf91e63c-e756-4218-8de0-8a22a4664b24","Type":"ContainerStarted","Data":"ad478dfd63c713b74863c6cc72358c7aa90f0baef68681b9bd1da87600131698"} Nov 03 00:00:01.321993 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:01.321960 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9" event={"ID":"9fad0b49-d23b-4913-b0fa-87e1c55c5a20","Type":"ContainerStarted","Data":"85370a7589739de4bb76854bdf88e5425ba924bd247c7a9b91cacbb52aef0c85"} Nov 03 00:00:01.324278 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:01.324251 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962" event={"ID":"e56f2e10-bb0e-474a-9611-7ef9d6d22cc7","Type":"ContainerStarted","Data":"4d6abca5f5eaba65752da032e9fed7752eac98ab1e81be920107ae5aba721f12"} Nov 03 00:00:01.328832 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:01.328808 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29894400-htxcw" event={"ID":"ae3e4d01-001e-4493-b7ac-7cab5bde77a0","Type":"ContainerStarted","Data":"3c264111bd71e427bb2db0fdac76625f1e6990b5c2292f4c7ad2eb7f97fc5623"} Nov 03 00:00:01.328922 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:01.328843 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29894400-htxcw" event={"ID":"ae3e4d01-001e-4493-b7ac-7cab5bde77a0","Type":"ContainerStarted","Data":"3d40a726df061fd8161a929142cba38e9c95f58f909886ea86a4dc1d3f391c3d"} Nov 03 00:00:01.443019 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:01.442949 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-pruner-29894400-htxcw" podStartSLOduration=1.442931511 podStartE2EDuration="1.442931511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:01.441976769 +0000 UTC m=+4287.680474443" watchObservedRunningTime="2026-11-03 00:00:01.442931511 +0000 UTC m=+4287.681429174" Nov 03 00:00:02.338212 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:02.338177 5727 generic.go:356] "Generic (PLEG): container finished" podID="c10cf5b1-d301-48d1-89a5-cac200b396d7" containerID="dc7e404f707d9f0a8544ee75af89b6f794102ba14166cb14023d78e9eb94edb5" exitCode=60 Nov 03 00:00:02.338631 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:02.338216 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7" event={"ID":"c10cf5b1-d301-48d1-89a5-cac200b396d7","Type":"ContainerDied","Data":"dc7e404f707d9f0a8544ee75af89b6f794102ba14166cb14023d78e9eb94edb5"} Nov 03 00:00:02.338631 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:02.338538 5727 scope.go:122] "RemoveContainer" containerID="dc7e404f707d9f0a8544ee75af89b6f794102ba14166cb14023d78e9eb94edb5" Nov 03 00:00:02.340191 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:02.340166 5727 generic.go:356] "Generic (PLEG): container finished" podID="bf91e63c-e756-4218-8de0-8a22a4664b24" containerID="93a77df2726f718ea8715cf238ac5f105cc93c083f9e46b9e3a29a01eb7524e0" exitCode=60 Nov 03 00:00:02.340381 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:02.340253 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt" event={"ID":"bf91e63c-e756-4218-8de0-8a22a4664b24","Type":"ContainerDied","Data":"93a77df2726f718ea8715cf238ac5f105cc93c083f9e46b9e3a29a01eb7524e0"} Nov 03 00:00:02.340524 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:02.340504 5727 scope.go:122] "RemoveContainer" containerID="93a77df2726f718ea8715cf238ac5f105cc93c083f9e46b9e3a29a01eb7524e0" Nov 03 00:00:02.342655 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:02.342628 5727 generic.go:356] "Generic (PLEG): container finished" podID="9fad0b49-d23b-4913-b0fa-87e1c55c5a20" containerID="dd078ab5faf858d7a420f646f05c69306e8e1675fd4652651ee3f99c63f2439b" exitCode=60 Nov 03 00:00:02.342759 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:02.342724 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9" event={"ID":"9fad0b49-d23b-4913-b0fa-87e1c55c5a20","Type":"ContainerDied","Data":"dd078ab5faf858d7a420f646f05c69306e8e1675fd4652651ee3f99c63f2439b"} Nov 03 00:00:02.342974 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:02.342958 5727 scope.go:122] "RemoveContainer" containerID="dd078ab5faf858d7a420f646f05c69306e8e1675fd4652651ee3f99c63f2439b" Nov 03 00:00:02.344889 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:02.344865 5727 generic.go:356] "Generic (PLEG): container finished" podID="e56f2e10-bb0e-474a-9611-7ef9d6d22cc7" containerID="1670b98a8dda6992feb26ca73f661af87b6df33a09c819ffe84ebc1e8ea8e992" exitCode=60 Nov 03 00:00:02.345098 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:02.345069 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962" event={"ID":"e56f2e10-bb0e-474a-9611-7ef9d6d22cc7","Type":"ContainerDied","Data":"1670b98a8dda6992feb26ca73f661af87b6df33a09c819ffe84ebc1e8ea8e992"} Nov 03 00:00:02.345358 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:02.345337 5727 scope.go:122] "RemoveContainer" containerID="1670b98a8dda6992feb26ca73f661af87b6df33a09c819ffe84ebc1e8ea8e992" Nov 03 00:00:03.356192 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:03.355735 5727 generic.go:356] "Generic (PLEG): container finished" podID="ae3e4d01-001e-4493-b7ac-7cab5bde77a0" containerID="3c264111bd71e427bb2db0fdac76625f1e6990b5c2292f4c7ad2eb7f97fc5623" exitCode=0 Nov 03 00:00:03.356192 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:03.355974 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29894400-htxcw" event={"ID":"ae3e4d01-001e-4493-b7ac-7cab5bde77a0","Type":"ContainerDied","Data":"3c264111bd71e427bb2db0fdac76625f1e6990b5c2292f4c7ad2eb7f97fc5623"} Nov 03 00:00:03.358767 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:03.358734 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7" event={"ID":"c10cf5b1-d301-48d1-89a5-cac200b396d7","Type":"ContainerStarted","Data":"88ec24201a82072bfdac56cc2b8a25f35b08c9f77b5a0d269d2eaa8c5fd10ffa"} Nov 03 00:00:03.361788 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:03.361761 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt" event={"ID":"bf91e63c-e756-4218-8de0-8a22a4664b24","Type":"ContainerStarted","Data":"cdfa53b6a21219437e43d70992dc3ee5836422140af542c00363d401653b1591"} Nov 03 00:00:03.365087 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:03.365054 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9" event={"ID":"9fad0b49-d23b-4913-b0fa-87e1c55c5a20","Type":"ContainerStarted","Data":"2b2aa9f5f13eb197975aa30f309aa985e729d1842fab240a249fa8511fc0849f"} Nov 03 00:00:03.367446 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:03.367409 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962" event={"ID":"e56f2e10-bb0e-474a-9611-7ef9d6d22cc7","Type":"ContainerStarted","Data":"eb119d5ade655fc573c50eaa8163bb16e82a6b6caebe3d62db60ba5e2d65790c"} Nov 03 00:00:03.588598 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:03.588506 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7" podStartSLOduration=3.588487069 podStartE2EDuration="3.588487069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:03.584053548 +0000 UTC m=+4289.822551209" watchObservedRunningTime="2026-11-03 00:00:03.588487069 +0000 UTC m=+4289.826984732" Nov 03 00:00:03.700670 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:03.700615 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962" podStartSLOduration=3.7006006879999997 podStartE2EDuration="3.700600688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:03.697472062 +0000 UTC m=+4289.935969735" watchObservedRunningTime="2026-11-03 00:00:03.700600688 +0000 UTC m=+4289.939098350" Nov 03 00:00:03.816293 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:03.816229 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9" podStartSLOduration=3.816211082 podStartE2EDuration="3.816211082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:03.813705031 +0000 UTC m=+4290.052202696" watchObservedRunningTime="2026-11-03 00:00:03.816211082 +0000 UTC m=+4290.054708745" Nov 03 00:00:03.929761 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:03.929639 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt" podStartSLOduration=3.929620432 podStartE2EDuration="3.929620432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:03.92938714 +0000 UTC m=+4290.167884825" watchObservedRunningTime="2026-11-03 00:00:03.929620432 +0000 UTC m=+4290.168118094" Nov 03 00:00:04.376991 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:04.376954 5727 generic.go:356] "Generic (PLEG): container finished" podID="c10cf5b1-d301-48d1-89a5-cac200b396d7" containerID="88ec24201a82072bfdac56cc2b8a25f35b08c9f77b5a0d269d2eaa8c5fd10ffa" exitCode=60 Nov 03 00:00:04.377540 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:04.377028 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7" event={"ID":"c10cf5b1-d301-48d1-89a5-cac200b396d7","Type":"ContainerDied","Data":"88ec24201a82072bfdac56cc2b8a25f35b08c9f77b5a0d269d2eaa8c5fd10ffa"} Nov 03 00:00:04.377540 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:04.377086 5727 scope.go:122] "RemoveContainer" containerID="dc7e404f707d9f0a8544ee75af89b6f794102ba14166cb14023d78e9eb94edb5" Nov 03 00:00:04.377540 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:04.377416 5727 scope.go:122] "RemoveContainer" containerID="88ec24201a82072bfdac56cc2b8a25f35b08c9f77b5a0d269d2eaa8c5fd10ffa" Nov 03 00:00:04.377784 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:00:04.377761 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7_odh-ai-gateway-infra(c10cf5b1-d301-48d1-89a5-cac200b396d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7" podUID="c10cf5b1-d301-48d1-89a5-cac200b396d7" Nov 03 00:00:04.379237 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:04.379209 5727 generic.go:356] "Generic (PLEG): container finished" podID="bf91e63c-e756-4218-8de0-8a22a4664b24" containerID="cdfa53b6a21219437e43d70992dc3ee5836422140af542c00363d401653b1591" exitCode=60 Nov 03 00:00:04.379342 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:04.379294 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt" event={"ID":"bf91e63c-e756-4218-8de0-8a22a4664b24","Type":"ContainerDied","Data":"cdfa53b6a21219437e43d70992dc3ee5836422140af542c00363d401653b1591"} Nov 03 00:00:04.379542 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:04.379528 5727 scope.go:122] "RemoveContainer" containerID="cdfa53b6a21219437e43d70992dc3ee5836422140af542c00363d401653b1591" Nov 03 00:00:04.379824 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:00:04.379797 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt_odh-ai-gateway-infra(bf91e63c-e756-4218-8de0-8a22a4664b24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt" podUID="bf91e63c-e756-4218-8de0-8a22a4664b24" Nov 03 00:00:04.381417 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:04.381397 5727 generic.go:356] "Generic (PLEG): container finished" podID="9fad0b49-d23b-4913-b0fa-87e1c55c5a20" containerID="2b2aa9f5f13eb197975aa30f309aa985e729d1842fab240a249fa8511fc0849f" exitCode=60 Nov 03 00:00:04.381514 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:04.381449 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9" event={"ID":"9fad0b49-d23b-4913-b0fa-87e1c55c5a20","Type":"ContainerDied","Data":"2b2aa9f5f13eb197975aa30f309aa985e729d1842fab240a249fa8511fc0849f"} Nov 03 00:00:04.381745 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:04.381726 5727 scope.go:122] "RemoveContainer" containerID="2b2aa9f5f13eb197975aa30f309aa985e729d1842fab240a249fa8511fc0849f" Nov 03 00:00:04.382044 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:00:04.382023 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9_odh-ai-gateway-infra(9fad0b49-d23b-4913-b0fa-87e1c55c5a20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9" podUID="9fad0b49-d23b-4913-b0fa-87e1c55c5a20" Nov 03 00:00:04.383486 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:04.383462 5727 generic.go:356] "Generic (PLEG): container finished" podID="e56f2e10-bb0e-474a-9611-7ef9d6d22cc7" containerID="eb119d5ade655fc573c50eaa8163bb16e82a6b6caebe3d62db60ba5e2d65790c" exitCode=60 Nov 03 00:00:04.383602 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:04.383528 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962" event={"ID":"e56f2e10-bb0e-474a-9611-7ef9d6d22cc7","Type":"ContainerDied","Data":"eb119d5ade655fc573c50eaa8163bb16e82a6b6caebe3d62db60ba5e2d65790c"} Nov 03 00:00:04.383797 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:04.383781 5727 scope.go:122] "RemoveContainer" containerID="eb119d5ade655fc573c50eaa8163bb16e82a6b6caebe3d62db60ba5e2d65790c" Nov 03 00:00:04.384100 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:00:04.384078 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962_odh-ai-gateway-infra(e56f2e10-bb0e-474a-9611-7ef9d6d22cc7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962" podUID="e56f2e10-bb0e-474a-9611-7ef9d6d22cc7" Nov 03 00:00:04.409252 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:04.409197 5727 scope.go:122] "RemoveContainer" containerID="93a77df2726f718ea8715cf238ac5f105cc93c083f9e46b9e3a29a01eb7524e0" Nov 03 00:00:04.426169 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:04.426108 5727 scope.go:122] "RemoveContainer" containerID="dd078ab5faf858d7a420f646f05c69306e8e1675fd4652651ee3f99c63f2439b" Nov 03 00:00:04.441938 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:04.441919 5727 scope.go:122] "RemoveContainer" containerID="1670b98a8dda6992feb26ca73f661af87b6df33a09c819ffe84ebc1e8ea8e992" Nov 03 00:00:04.567666 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:04.567639 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29894400-htxcw" Nov 03 00:00:04.647256 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:04.647179 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae3e4d01-001e-4493-b7ac-7cab5bde77a0-kube-api-access-nc2n9\" (UniqueName: \"kubernetes.io/projected/ae3e4d01-001e-4493-b7ac-7cab5bde77a0-kube-api-access-nc2n9\") pod \"ae3e4d01-001e-4493-b7ac-7cab5bde77a0\" (UID: \"ae3e4d01-001e-4493-b7ac-7cab5bde77a0\") " Nov 03 00:00:04.647256 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:04.647253 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae3e4d01-001e-4493-b7ac-7cab5bde77a0-serviceca\" (UniqueName: \"kubernetes.io/configmap/ae3e4d01-001e-4493-b7ac-7cab5bde77a0-serviceca\") pod \"ae3e4d01-001e-4493-b7ac-7cab5bde77a0\" (UID: \"ae3e4d01-001e-4493-b7ac-7cab5bde77a0\") " Nov 03 00:00:04.647634 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:04.647607 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae3e4d01-001e-4493-b7ac-7cab5bde77a0-serviceca" pod "ae3e4d01-001e-4493-b7ac-7cab5bde77a0" (UID: "ae3e4d01-001e-4493-b7ac-7cab5bde77a0"). InnerVolumeSpecName "serviceca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:00:04.647814 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:04.647800 5727 reconciler_common.go:299] "Volume detached for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ae3e4d01-001e-4493-b7ac-7cab5bde77a0-serviceca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:04.649457 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:04.649437 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae3e4d01-001e-4493-b7ac-7cab5bde77a0-kube-api-access-nc2n9" pod "ae3e4d01-001e-4493-b7ac-7cab5bde77a0" (UID: "ae3e4d01-001e-4493-b7ac-7cab5bde77a0"). InnerVolumeSpecName "kube-api-access-nc2n9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:04.749509 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:04.749479 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc2n9\" (UniqueName: \"kubernetes.io/projected/ae3e4d01-001e-4493-b7ac-7cab5bde77a0-kube-api-access-nc2n9\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:05.393763 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:05.393732 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29894400-htxcw" Nov 03 00:00:05.393763 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:05.393750 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29894400-htxcw" event={"ID":"ae3e4d01-001e-4493-b7ac-7cab5bde77a0","Type":"ContainerDied","Data":"3d40a726df061fd8161a929142cba38e9c95f58f909886ea86a4dc1d3f391c3d"} Nov 03 00:00:05.394384 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:05.393800 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d40a726df061fd8161a929142cba38e9c95f58f909886ea86a4dc1d3f391c3d" Nov 03 00:00:05.396498 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:05.396471 5727 scope.go:122] "RemoveContainer" containerID="88ec24201a82072bfdac56cc2b8a25f35b08c9f77b5a0d269d2eaa8c5fd10ffa" Nov 03 00:00:05.396981 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:00:05.396799 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7_odh-ai-gateway-infra(c10cf5b1-d301-48d1-89a5-cac200b396d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7" podUID="c10cf5b1-d301-48d1-89a5-cac200b396d7" Nov 03 00:00:05.398738 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:05.398716 5727 scope.go:122] "RemoveContainer" containerID="cdfa53b6a21219437e43d70992dc3ee5836422140af542c00363d401653b1591" Nov 03 00:00:05.399015 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:00:05.398992 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt_odh-ai-gateway-infra(bf91e63c-e756-4218-8de0-8a22a4664b24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt" podUID="bf91e63c-e756-4218-8de0-8a22a4664b24" Nov 03 00:00:05.400778 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:05.400754 5727 scope.go:122] "RemoveContainer" containerID="2b2aa9f5f13eb197975aa30f309aa985e729d1842fab240a249fa8511fc0849f" Nov 03 00:00:05.401048 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:00:05.401022 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9_odh-ai-gateway-infra(9fad0b49-d23b-4913-b0fa-87e1c55c5a20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9" podUID="9fad0b49-d23b-4913-b0fa-87e1c55c5a20" Nov 03 00:00:05.402773 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:05.402750 5727 scope.go:122] "RemoveContainer" containerID="eb119d5ade655fc573c50eaa8163bb16e82a6b6caebe3d62db60ba5e2d65790c" Nov 03 00:00:05.403001 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:00:05.402977 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962_odh-ai-gateway-infra(e56f2e10-bb0e-474a-9611-7ef9d6d22cc7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962" podUID="e56f2e10-bb0e-474a-9611-7ef9d6d22cc7" Nov 03 00:00:15.876029 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:15.875997 5727 scope.go:122] "RemoveContainer" containerID="2b2aa9f5f13eb197975aa30f309aa985e729d1842fab240a249fa8511fc0849f" Nov 03 00:00:16.515912 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:16.515826 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9" event={"ID":"9fad0b49-d23b-4913-b0fa-87e1c55c5a20","Type":"ContainerStarted","Data":"67d822882df3dfdb9a2c3975d4f8fe69876bbed22bda98a6f32fbc1ce623a879"} Nov 03 00:00:16.875837 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:16.875808 5727 scope.go:122] "RemoveContainer" containerID="cdfa53b6a21219437e43d70992dc3ee5836422140af542c00363d401653b1591" Nov 03 00:00:16.875954 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:16.875867 5727 scope.go:122] "RemoveContainer" containerID="88ec24201a82072bfdac56cc2b8a25f35b08c9f77b5a0d269d2eaa8c5fd10ffa" Nov 03 00:00:16.876039 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:16.876014 5727 scope.go:122] "RemoveContainer" containerID="eb119d5ade655fc573c50eaa8163bb16e82a6b6caebe3d62db60ba5e2d65790c" Nov 03 00:00:17.525697 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:17.525612 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7" event={"ID":"c10cf5b1-d301-48d1-89a5-cac200b396d7","Type":"ContainerStarted","Data":"163e2eccdac64eb172d14e943425d5c148d181221bb4000c1dad9aa70564304b"} Nov 03 00:00:17.527722 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:17.527694 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt" event={"ID":"bf91e63c-e756-4218-8de0-8a22a4664b24","Type":"ContainerStarted","Data":"06ffb03fecfb295b51f0213ff06bc6e2d100bc8a08f365b4d320bb37b61e4408"} Nov 03 00:00:17.529546 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:17.529521 5727 generic.go:356] "Generic (PLEG): container finished" podID="9fad0b49-d23b-4913-b0fa-87e1c55c5a20" containerID="67d822882df3dfdb9a2c3975d4f8fe69876bbed22bda98a6f32fbc1ce623a879" exitCode=60 Nov 03 00:00:17.529630 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:17.529594 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9" event={"ID":"9fad0b49-d23b-4913-b0fa-87e1c55c5a20","Type":"ContainerDied","Data":"67d822882df3dfdb9a2c3975d4f8fe69876bbed22bda98a6f32fbc1ce623a879"} Nov 03 00:00:17.529691 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:17.529647 5727 scope.go:122] "RemoveContainer" containerID="2b2aa9f5f13eb197975aa30f309aa985e729d1842fab240a249fa8511fc0849f" Nov 03 00:00:17.529977 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:17.529954 5727 scope.go:122] "RemoveContainer" containerID="67d822882df3dfdb9a2c3975d4f8fe69876bbed22bda98a6f32fbc1ce623a879" Nov 03 00:00:17.530280 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:00:17.530257 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9_odh-ai-gateway-infra(9fad0b49-d23b-4913-b0fa-87e1c55c5a20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9" podUID="9fad0b49-d23b-4913-b0fa-87e1c55c5a20" Nov 03 00:00:17.531702 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:17.531682 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962" event={"ID":"e56f2e10-bb0e-474a-9611-7ef9d6d22cc7","Type":"ContainerStarted","Data":"0c783d6b19fdecc0dc1cdcfc9ef4cbd5e70699239098d468dc4a06adf2f8d431"} Nov 03 00:00:17.641442 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:17.641412 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9"] Nov 03 00:00:18.552442 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:18.552410 5727 generic.go:356] "Generic (PLEG): container finished" podID="c10cf5b1-d301-48d1-89a5-cac200b396d7" containerID="163e2eccdac64eb172d14e943425d5c148d181221bb4000c1dad9aa70564304b" exitCode=60 Nov 03 00:00:18.552958 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:18.552484 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7" event={"ID":"c10cf5b1-d301-48d1-89a5-cac200b396d7","Type":"ContainerDied","Data":"163e2eccdac64eb172d14e943425d5c148d181221bb4000c1dad9aa70564304b"} Nov 03 00:00:18.552958 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:18.552593 5727 scope.go:122] "RemoveContainer" containerID="88ec24201a82072bfdac56cc2b8a25f35b08c9f77b5a0d269d2eaa8c5fd10ffa" Nov 03 00:00:18.553197 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:18.553117 5727 scope.go:122] "RemoveContainer" containerID="163e2eccdac64eb172d14e943425d5c148d181221bb4000c1dad9aa70564304b" Nov 03 00:00:18.553631 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:00:18.553607 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7_odh-ai-gateway-infra(c10cf5b1-d301-48d1-89a5-cac200b396d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7" podUID="c10cf5b1-d301-48d1-89a5-cac200b396d7" Nov 03 00:00:18.555224 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:18.555202 5727 generic.go:356] "Generic (PLEG): container finished" podID="bf91e63c-e756-4218-8de0-8a22a4664b24" containerID="06ffb03fecfb295b51f0213ff06bc6e2d100bc8a08f365b4d320bb37b61e4408" exitCode=60 Nov 03 00:00:18.555342 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:18.555266 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt" event={"ID":"bf91e63c-e756-4218-8de0-8a22a4664b24","Type":"ContainerDied","Data":"06ffb03fecfb295b51f0213ff06bc6e2d100bc8a08f365b4d320bb37b61e4408"} Nov 03 00:00:18.555622 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:18.555605 5727 scope.go:122] "RemoveContainer" containerID="06ffb03fecfb295b51f0213ff06bc6e2d100bc8a08f365b4d320bb37b61e4408" Nov 03 00:00:18.555910 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:00:18.555889 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt_odh-ai-gateway-infra(bf91e63c-e756-4218-8de0-8a22a4664b24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt" podUID="bf91e63c-e756-4218-8de0-8a22a4664b24" Nov 03 00:00:18.584180 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:18.573928 5727 generic.go:356] "Generic (PLEG): container finished" podID="e56f2e10-bb0e-474a-9611-7ef9d6d22cc7" containerID="0c783d6b19fdecc0dc1cdcfc9ef4cbd5e70699239098d468dc4a06adf2f8d431" exitCode=60 Nov 03 00:00:18.584180 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:18.574112 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962" event={"ID":"e56f2e10-bb0e-474a-9611-7ef9d6d22cc7","Type":"ContainerDied","Data":"0c783d6b19fdecc0dc1cdcfc9ef4cbd5e70699239098d468dc4a06adf2f8d431"} Nov 03 00:00:18.584180 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:18.574561 5727 scope.go:122] "RemoveContainer" containerID="0c783d6b19fdecc0dc1cdcfc9ef4cbd5e70699239098d468dc4a06adf2f8d431" Nov 03 00:00:18.584180 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:00:18.574894 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962_odh-ai-gateway-infra(e56f2e10-bb0e-474a-9611-7ef9d6d22cc7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962" podUID="e56f2e10-bb0e-474a-9611-7ef9d6d22cc7" Nov 03 00:00:18.584180 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:18.580517 5727 scope.go:122] "RemoveContainer" containerID="cdfa53b6a21219437e43d70992dc3ee5836422140af542c00363d401653b1591" Nov 03 00:00:18.597504 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:18.597474 5727 scope.go:122] "RemoveContainer" containerID="eb119d5ade655fc573c50eaa8163bb16e82a6b6caebe3d62db60ba5e2d65790c" Nov 03 00:00:18.651662 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:18.651627 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7"] Nov 03 00:00:18.739189 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:18.739166 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9" Nov 03 00:00:18.766094 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:18.766065 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962"] Nov 03 00:00:18.782792 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:18.782767 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fad0b49-d23b-4913-b0fa-87e1c55c5a20-kube-api-access-xz8nc\" (UniqueName: \"kubernetes.io/projected/9fad0b49-d23b-4913-b0fa-87e1c55c5a20-kube-api-access-xz8nc\") pod \"9fad0b49-d23b-4913-b0fa-87e1c55c5a20\" (UID: \"9fad0b49-d23b-4913-b0fa-87e1c55c5a20\") " Nov 03 00:00:18.783200 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:18.782895 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fad0b49-d23b-4913-b0fa-87e1c55c5a20-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fad0b49-d23b-4913-b0fa-87e1c55c5a20-maas-api-service-ca\") pod \"9fad0b49-d23b-4913-b0fa-87e1c55c5a20\" (UID: \"9fad0b49-d23b-4913-b0fa-87e1c55c5a20\") " Nov 03 00:00:18.783592 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:18.783566 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fad0b49-d23b-4913-b0fa-87e1c55c5a20-maas-api-service-ca" pod "9fad0b49-d23b-4913-b0fa-87e1c55c5a20" (UID: "9fad0b49-d23b-4913-b0fa-87e1c55c5a20"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:00:18.786167 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:18.786115 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fad0b49-d23b-4913-b0fa-87e1c55c5a20-kube-api-access-xz8nc" pod "9fad0b49-d23b-4913-b0fa-87e1c55c5a20" (UID: "9fad0b49-d23b-4913-b0fa-87e1c55c5a20"). InnerVolumeSpecName "kube-api-access-xz8nc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:18.884646 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:18.884614 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz8nc\" (UniqueName: \"kubernetes.io/projected/9fad0b49-d23b-4913-b0fa-87e1c55c5a20-kube-api-access-xz8nc\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:18.884646 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:18.884640 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fad0b49-d23b-4913-b0fa-87e1c55c5a20-maas-api-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:19.001788 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:19.001752 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt"] Nov 03 00:00:19.590214 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:19.590178 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9" event={"ID":"9fad0b49-d23b-4913-b0fa-87e1c55c5a20","Type":"ContainerDied","Data":"85370a7589739de4bb76854bdf88e5425ba924bd247c7a9b91cacbb52aef0c85"} Nov 03 00:00:19.590737 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:19.590715 5727 scope.go:122] "RemoveContainer" containerID="67d822882df3dfdb9a2c3975d4f8fe69876bbed22bda98a6f32fbc1ce623a879" Nov 03 00:00:19.590907 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:19.590357 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9" Nov 03 00:00:19.791911 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:19.791388 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7" Nov 03 00:00:19.798798 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:19.798759 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c10cf5b1-d301-48d1-89a5-cac200b396d7-kube-api-access-hvr4l\" (UniqueName: \"kubernetes.io/projected/c10cf5b1-d301-48d1-89a5-cac200b396d7-kube-api-access-hvr4l\") pod \"c10cf5b1-d301-48d1-89a5-cac200b396d7\" (UID: \"c10cf5b1-d301-48d1-89a5-cac200b396d7\") " Nov 03 00:00:19.799349 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:19.799321 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c10cf5b1-d301-48d1-89a5-cac200b396d7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c10cf5b1-d301-48d1-89a5-cac200b396d7-maas-api-service-ca\") pod \"c10cf5b1-d301-48d1-89a5-cac200b396d7\" (UID: \"c10cf5b1-d301-48d1-89a5-cac200b396d7\") " Nov 03 00:00:19.799709 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:19.799685 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c10cf5b1-d301-48d1-89a5-cac200b396d7-maas-api-service-ca" pod "c10cf5b1-d301-48d1-89a5-cac200b396d7" (UID: "c10cf5b1-d301-48d1-89a5-cac200b396d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:00:19.800248 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:19.800224 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c10cf5b1-d301-48d1-89a5-cac200b396d7-maas-api-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:19.802633 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:19.802596 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c10cf5b1-d301-48d1-89a5-cac200b396d7-kube-api-access-hvr4l" pod "c10cf5b1-d301-48d1-89a5-cac200b396d7" (UID: "c10cf5b1-d301-48d1-89a5-cac200b396d7"). InnerVolumeSpecName "kube-api-access-hvr4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:19.859745 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:19.859723 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962" Nov 03 00:00:19.863685 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:19.863666 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt" Nov 03 00:00:19.900863 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:19.900839 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf91e63c-e756-4218-8de0-8a22a4664b24-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf91e63c-e756-4218-8de0-8a22a4664b24-maas-api-service-ca\") pod \"bf91e63c-e756-4218-8de0-8a22a4664b24\" (UID: \"bf91e63c-e756-4218-8de0-8a22a4664b24\") " Nov 03 00:00:19.900972 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:19.900911 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e56f2e10-bb0e-474a-9611-7ef9d6d22cc7-kube-api-access-qccdb\" (UniqueName: \"kubernetes.io/projected/e56f2e10-bb0e-474a-9611-7ef9d6d22cc7-kube-api-access-qccdb\") pod \"e56f2e10-bb0e-474a-9611-7ef9d6d22cc7\" (UID: \"e56f2e10-bb0e-474a-9611-7ef9d6d22cc7\") " Nov 03 00:00:19.901042 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:19.900980 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf91e63c-e756-4218-8de0-8a22a4664b24-kube-api-access-jbl4d\" (UniqueName: \"kubernetes.io/projected/bf91e63c-e756-4218-8de0-8a22a4664b24-kube-api-access-jbl4d\") pod \"bf91e63c-e756-4218-8de0-8a22a4664b24\" (UID: \"bf91e63c-e756-4218-8de0-8a22a4664b24\") " Nov 03 00:00:19.901105 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:19.901088 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e56f2e10-bb0e-474a-9611-7ef9d6d22cc7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e56f2e10-bb0e-474a-9611-7ef9d6d22cc7-maas-api-service-ca\") pod \"e56f2e10-bb0e-474a-9611-7ef9d6d22cc7\" (UID: \"e56f2e10-bb0e-474a-9611-7ef9d6d22cc7\") " Nov 03 00:00:19.901186 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:19.901166 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf91e63c-e756-4218-8de0-8a22a4664b24-maas-api-service-ca" pod "bf91e63c-e756-4218-8de0-8a22a4664b24" (UID: "bf91e63c-e756-4218-8de0-8a22a4664b24"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:00:19.901428 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:19.901404 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e56f2e10-bb0e-474a-9611-7ef9d6d22cc7-maas-api-service-ca" pod "e56f2e10-bb0e-474a-9611-7ef9d6d22cc7" (UID: "e56f2e10-bb0e-474a-9611-7ef9d6d22cc7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:00:19.901953 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:19.901917 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e56f2e10-bb0e-474a-9611-7ef9d6d22cc7-maas-api-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:19.902030 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:19.901960 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf91e63c-e756-4218-8de0-8a22a4664b24-maas-api-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:19.902030 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:19.901978 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvr4l\" (UniqueName: \"kubernetes.io/projected/c10cf5b1-d301-48d1-89a5-cac200b396d7-kube-api-access-hvr4l\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:19.903321 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:19.903300 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e56f2e10-bb0e-474a-9611-7ef9d6d22cc7-kube-api-access-qccdb" pod "e56f2e10-bb0e-474a-9611-7ef9d6d22cc7" (UID: "e56f2e10-bb0e-474a-9611-7ef9d6d22cc7"). InnerVolumeSpecName "kube-api-access-qccdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:19.903394 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:19.903351 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf91e63c-e756-4218-8de0-8a22a4664b24-kube-api-access-jbl4d" pod "bf91e63c-e756-4218-8de0-8a22a4664b24" (UID: "bf91e63c-e756-4218-8de0-8a22a4664b24"). InnerVolumeSpecName "kube-api-access-jbl4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:20.003492 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:20.003451 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qccdb\" (UniqueName: \"kubernetes.io/projected/e56f2e10-bb0e-474a-9611-7ef9d6d22cc7-kube-api-access-qccdb\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:20.003492 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:20.003484 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbl4d\" (UniqueName: \"kubernetes.io/projected/bf91e63c-e756-4218-8de0-8a22a4664b24-kube-api-access-jbl4d\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:20.053968 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:20.053560 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9"] Nov 03 00:00:20.059473 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:20.059443 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894400-4jxf9"] Nov 03 00:00:20.606381 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:20.606351 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962" Nov 03 00:00:20.606381 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:20.606358 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962" event={"ID":"e56f2e10-bb0e-474a-9611-7ef9d6d22cc7","Type":"ContainerDied","Data":"4d6abca5f5eaba65752da032e9fed7752eac98ab1e81be920107ae5aba721f12"} Nov 03 00:00:20.606937 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:20.606424 5727 scope.go:122] "RemoveContainer" containerID="0c783d6b19fdecc0dc1cdcfc9ef4cbd5e70699239098d468dc4a06adf2f8d431" Nov 03 00:00:20.608090 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:20.608070 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7" Nov 03 00:00:20.608090 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:20.608081 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7" event={"ID":"c10cf5b1-d301-48d1-89a5-cac200b396d7","Type":"ContainerDied","Data":"9ea770bdf567c97b5545a0bd405ea8cd56cbeffbdc12b7717afbbfefca7475c1"} Nov 03 00:00:20.609769 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:20.609732 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt" event={"ID":"bf91e63c-e756-4218-8de0-8a22a4664b24","Type":"ContainerDied","Data":"ad478dfd63c713b74863c6cc72358c7aa90f0baef68681b9bd1da87600131698"} Nov 03 00:00:20.609769 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:20.609767 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt" Nov 03 00:00:20.619925 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:20.619908 5727 scope.go:122] "RemoveContainer" containerID="163e2eccdac64eb172d14e943425d5c148d181221bb4000c1dad9aa70564304b" Nov 03 00:00:20.631337 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:20.631319 5727 scope.go:122] "RemoveContainer" containerID="06ffb03fecfb295b51f0213ff06bc6e2d100bc8a08f365b4d320bb37b61e4408" Nov 03 00:00:20.772409 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:20.772373 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7"] Nov 03 00:00:20.821442 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:20.821405 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894400-64qg7"] Nov 03 00:00:20.882213 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:20.882121 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962"] Nov 03 00:00:20.887491 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:20.887463 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894400-dz962"] Nov 03 00:00:20.995929 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:20.995887 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt"] Nov 03 00:00:20.999105 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:20.999076 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894400-wkbmt"] Nov 03 00:00:21.885337 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:21.885297 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fad0b49-d23b-4913-b0fa-87e1c55c5a20" path="/var/lib/kubelet/pods/9fad0b49-d23b-4913-b0fa-87e1c55c5a20/volumes" Nov 03 00:00:21.885817 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:21.885706 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf91e63c-e756-4218-8de0-8a22a4664b24" path="/var/lib/kubelet/pods/bf91e63c-e756-4218-8de0-8a22a4664b24/volumes" Nov 03 00:00:21.886063 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:21.886049 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c10cf5b1-d301-48d1-89a5-cac200b396d7" path="/var/lib/kubelet/pods/c10cf5b1-d301-48d1-89a5-cac200b396d7/volumes" Nov 03 00:00:21.886743 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:00:21.886726 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e56f2e10-bb0e-474a-9611-7ef9d6d22cc7" path="/var/lib/kubelet/pods/e56f2e10-bb0e-474a-9611-7ef9d6d22cc7/volumes" Nov 03 00:00:27.846154 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:00:27.846095 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:00:57.845633 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:00:57.845590 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:01:10.682839 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.682802 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bee1b-64dd49579f-7psbr"] Nov 03 00:01:10.683516 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683469 5727 cpu_manager.go:414] "containerMap: removing container" podUID="e56f2e10-bb0e-474a-9611-7ef9d6d22cc7" containerName="cleanup" Nov 03 00:01:10.683516 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683486 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e56f2e10-bb0e-474a-9611-7ef9d6d22cc7" containerName="cleanup" Nov 03 00:01:10.683516 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683500 5727 cpu_manager.go:414] "containerMap: removing container" podUID="e56f2e10-bb0e-474a-9611-7ef9d6d22cc7" containerName="cleanup" Nov 03 00:01:10.683516 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683505 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e56f2e10-bb0e-474a-9611-7ef9d6d22cc7" containerName="cleanup" Nov 03 00:01:10.683516 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683517 5727 cpu_manager.go:414] "containerMap: removing container" podUID="e56f2e10-bb0e-474a-9611-7ef9d6d22cc7" containerName="cleanup" Nov 03 00:01:10.683780 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683524 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e56f2e10-bb0e-474a-9611-7ef9d6d22cc7" containerName="cleanup" Nov 03 00:01:10.683780 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683537 5727 cpu_manager.go:414] "containerMap: removing container" podUID="c10cf5b1-d301-48d1-89a5-cac200b396d7" containerName="cleanup" Nov 03 00:01:10.683780 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683544 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c10cf5b1-d301-48d1-89a5-cac200b396d7" containerName="cleanup" Nov 03 00:01:10.683780 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683563 5727 cpu_manager.go:414] "containerMap: removing container" podUID="bf91e63c-e756-4218-8de0-8a22a4664b24" containerName="cleanup" Nov 03 00:01:10.683780 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683570 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf91e63c-e756-4218-8de0-8a22a4664b24" containerName="cleanup" Nov 03 00:01:10.683780 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683581 5727 cpu_manager.go:414] "containerMap: removing container" podUID="bf91e63c-e756-4218-8de0-8a22a4664b24" containerName="cleanup" Nov 03 00:01:10.683780 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683587 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf91e63c-e756-4218-8de0-8a22a4664b24" containerName="cleanup" Nov 03 00:01:10.683780 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683613 5727 cpu_manager.go:414] "containerMap: removing container" podUID="ae3e4d01-001e-4493-b7ac-7cab5bde77a0" containerName="image-pruner" Nov 03 00:01:10.683780 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683620 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae3e4d01-001e-4493-b7ac-7cab5bde77a0" containerName="image-pruner" Nov 03 00:01:10.683780 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683641 5727 cpu_manager.go:414] "containerMap: removing container" podUID="9fad0b49-d23b-4913-b0fa-87e1c55c5a20" containerName="cleanup" Nov 03 00:01:10.683780 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683646 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fad0b49-d23b-4913-b0fa-87e1c55c5a20" containerName="cleanup" Nov 03 00:01:10.683780 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683654 5727 cpu_manager.go:414] "containerMap: removing container" podUID="9fad0b49-d23b-4913-b0fa-87e1c55c5a20" containerName="cleanup" Nov 03 00:01:10.683780 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683658 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fad0b49-d23b-4913-b0fa-87e1c55c5a20" containerName="cleanup" Nov 03 00:01:10.683780 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683682 5727 cpu_manager.go:414] "containerMap: removing container" podUID="9fad0b49-d23b-4913-b0fa-87e1c55c5a20" containerName="cleanup" Nov 03 00:01:10.683780 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683691 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fad0b49-d23b-4913-b0fa-87e1c55c5a20" containerName="cleanup" Nov 03 00:01:10.683780 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683713 5727 cpu_manager.go:414] "containerMap: removing container" podUID="c10cf5b1-d301-48d1-89a5-cac200b396d7" containerName="cleanup" Nov 03 00:01:10.683780 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683720 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c10cf5b1-d301-48d1-89a5-cac200b396d7" containerName="cleanup" Nov 03 00:01:10.684469 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683885 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae3e4d01-001e-4493-b7ac-7cab5bde77a0" containerName="image-pruner" Nov 03 00:01:10.684469 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683897 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fad0b49-d23b-4913-b0fa-87e1c55c5a20" containerName="cleanup" Nov 03 00:01:10.684469 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683908 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fad0b49-d23b-4913-b0fa-87e1c55c5a20" containerName="cleanup" Nov 03 00:01:10.684469 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683919 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="c10cf5b1-d301-48d1-89a5-cac200b396d7" containerName="cleanup" Nov 03 00:01:10.684469 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683928 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf91e63c-e756-4218-8de0-8a22a4664b24" containerName="cleanup" Nov 03 00:01:10.684469 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683938 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="e56f2e10-bb0e-474a-9611-7ef9d6d22cc7" containerName="cleanup" Nov 03 00:01:10.684469 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683944 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="e56f2e10-bb0e-474a-9611-7ef9d6d22cc7" containerName="cleanup" Nov 03 00:01:10.684469 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683951 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="e56f2e10-bb0e-474a-9611-7ef9d6d22cc7" containerName="cleanup" Nov 03 00:01:10.684469 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683955 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="c10cf5b1-d301-48d1-89a5-cac200b396d7" containerName="cleanup" Nov 03 00:01:10.684469 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683961 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf91e63c-e756-4218-8de0-8a22a4664b24" containerName="cleanup" Nov 03 00:01:10.684469 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.683966 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf91e63c-e756-4218-8de0-8a22a4664b24" containerName="cleanup" Nov 03 00:01:10.687842 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.687821 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bee1b-64dd49579f-7psbr" Nov 03 00:01:10.744580 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.744545 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bee1b-64dd49579f-7psbr"] Nov 03 00:01:10.763736 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.763676 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btm5x\" (UniqueName: \"kubernetes.io/projected/d6dcd3f9-d170-404b-9921-83b0d2e47ed4-kube-api-access-btm5x\") pod \"payload-pre-processing-e2e-shared-a-w1-3bee1b-64dd49579f-7psbr\" (UID: \"d6dcd3f9-d170-404b-9921-83b0d2e47ed4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bee1b-64dd49579f-7psbr" Nov 03 00:01:10.763941 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.763868 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6dcd3f9-d170-404b-9921-83b0d2e47ed4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-3bee1b-64dd49579f-7psbr\" (UID: \"d6dcd3f9-d170-404b-9921-83b0d2e47ed4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bee1b-64dd49579f-7psbr" Nov 03 00:01:10.790011 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.789973 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-3bee1b\"" Nov 03 00:01:10.845301 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.845266 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-3bee1b-5899c6f99f-8ftj2"] Nov 03 00:01:10.846194 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.846053 5727 cpu_manager.go:414] "containerMap: removing container" podUID="bf91e63c-e756-4218-8de0-8a22a4664b24" containerName="cleanup" Nov 03 00:01:10.846194 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.846074 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf91e63c-e756-4218-8de0-8a22a4664b24" containerName="cleanup" Nov 03 00:01:10.846194 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.846178 5727 cpu_manager.go:414] "containerMap: removing container" podUID="c10cf5b1-d301-48d1-89a5-cac200b396d7" containerName="cleanup" Nov 03 00:01:10.846194 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.846189 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c10cf5b1-d301-48d1-89a5-cac200b396d7" containerName="cleanup" Nov 03 00:01:10.846508 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.846357 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fad0b49-d23b-4913-b0fa-87e1c55c5a20" containerName="cleanup" Nov 03 00:01:10.846508 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.846383 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="c10cf5b1-d301-48d1-89a5-cac200b396d7" containerName="cleanup" Nov 03 00:01:10.850549 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.850203 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bee1b-5899c6f99f-8ftj2" Nov 03 00:01:10.866651 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.866592 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6dcd3f9-d170-404b-9921-83b0d2e47ed4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-3bee1b-64dd49579f-7psbr\" (UID: \"d6dcd3f9-d170-404b-9921-83b0d2e47ed4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bee1b-64dd49579f-7psbr" Nov 03 00:01:10.867061 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.867034 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btm5x\" (UniqueName: \"kubernetes.io/projected/d6dcd3f9-d170-404b-9921-83b0d2e47ed4-kube-api-access-btm5x\") pod \"payload-pre-processing-e2e-shared-a-w1-3bee1b-64dd49579f-7psbr\" (UID: \"d6dcd3f9-d170-404b-9921-83b0d2e47ed4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bee1b-64dd49579f-7psbr" Nov 03 00:01:10.868449 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.868420 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6dcd3f9-d170-404b-9921-83b0d2e47ed4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-3bee1b-64dd49579f-7psbr\" (UID: \"d6dcd3f9-d170-404b-9921-83b0d2e47ed4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bee1b-64dd49579f-7psbr" Nov 03 00:01:10.908150 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.904918 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-3bee1b-dockercfg-8vrtq\"" Nov 03 00:01:10.911217 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.910072 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-3bee1b-5899c6f99f-8ftj2"] Nov 03 00:01:10.928299 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.928271 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btm5x\" (UniqueName: \"kubernetes.io/projected/d6dcd3f9-d170-404b-9921-83b0d2e47ed4-kube-api-access-btm5x\") pod \"payload-pre-processing-e2e-shared-a-w1-3bee1b-64dd49579f-7psbr\" (UID: \"d6dcd3f9-d170-404b-9921-83b0d2e47ed4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bee1b-64dd49579f-7psbr" Nov 03 00:01:10.969105 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.969021 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/940cf7dc-acaa-4ff3-b473-99c7536913cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-3bee1b-5899c6f99f-8ftj2\" (UID: \"940cf7dc-acaa-4ff3-b473-99c7536913cf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bee1b-5899c6f99f-8ftj2" Nov 03 00:01:10.969282 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:10.969203 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8txq\" (UniqueName: \"kubernetes.io/projected/940cf7dc-acaa-4ff3-b473-99c7536913cf-kube-api-access-c8txq\") pod \"payload-processing-e2e-shared-a-w1-3bee1b-5899c6f99f-8ftj2\" (UID: \"940cf7dc-acaa-4ff3-b473-99c7536913cf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bee1b-5899c6f99f-8ftj2" Nov 03 00:01:11.004217 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:11.004191 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bee1b-64dd49579f-7psbr" Nov 03 00:01:11.070762 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:11.070711 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8txq\" (UniqueName: \"kubernetes.io/projected/940cf7dc-acaa-4ff3-b473-99c7536913cf-kube-api-access-c8txq\") pod \"payload-processing-e2e-shared-a-w1-3bee1b-5899c6f99f-8ftj2\" (UID: \"940cf7dc-acaa-4ff3-b473-99c7536913cf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bee1b-5899c6f99f-8ftj2" Nov 03 00:01:11.070944 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:11.070926 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/940cf7dc-acaa-4ff3-b473-99c7536913cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-3bee1b-5899c6f99f-8ftj2\" (UID: \"940cf7dc-acaa-4ff3-b473-99c7536913cf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bee1b-5899c6f99f-8ftj2" Nov 03 00:01:11.072060 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:11.072033 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/940cf7dc-acaa-4ff3-b473-99c7536913cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-3bee1b-5899c6f99f-8ftj2\" (UID: \"940cf7dc-acaa-4ff3-b473-99c7536913cf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bee1b-5899c6f99f-8ftj2" Nov 03 00:01:11.131515 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:11.131481 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8txq\" (UniqueName: \"kubernetes.io/projected/940cf7dc-acaa-4ff3-b473-99c7536913cf-kube-api-access-c8txq\") pod \"payload-processing-e2e-shared-a-w1-3bee1b-5899c6f99f-8ftj2\" (UID: \"940cf7dc-acaa-4ff3-b473-99c7536913cf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bee1b-5899c6f99f-8ftj2" Nov 03 00:01:11.200742 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:11.200710 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bee1b-5899c6f99f-8ftj2" Nov 03 00:01:11.389190 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:11.389159 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bee1b-64dd49579f-7psbr"] Nov 03 00:01:11.390718 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:01:11.390686 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6dcd3f9_d170_404b_9921_83b0d2e47ed4.slice/crio-34f3add97d1a94e4eb2df68d23bca4f2c38951ba74d8b81a46c74fc6ab24b013 WatchSource:0}: Error finding container 34f3add97d1a94e4eb2df68d23bca4f2c38951ba74d8b81a46c74fc6ab24b013: Status 404 returned error can't find the container with id 34f3add97d1a94e4eb2df68d23bca4f2c38951ba74d8b81a46c74fc6ab24b013 Nov 03 00:01:11.585541 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:11.585511 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-3bee1b-5899c6f99f-8ftj2"] Nov 03 00:01:11.587090 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:01:11.587063 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940cf7dc_acaa_4ff3_b473_99c7536913cf.slice/crio-62bc16fb099378296e5eddf5e3c6c3204139be052311df3db87e72ac411bde1d WatchSource:0}: Error finding container 62bc16fb099378296e5eddf5e3c6c3204139be052311df3db87e72ac411bde1d: Status 404 returned error can't find the container with id 62bc16fb099378296e5eddf5e3c6c3204139be052311df3db87e72ac411bde1d Nov 03 00:01:12.002038 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:12.001931 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bee1b-5899c6f99f-8ftj2" event={"ID":"940cf7dc-acaa-4ff3-b473-99c7536913cf","Type":"ContainerStarted","Data":"d4d844e36ae03f4a5627ac0c37488838b4c891597114b88e00cff75a8bedf262"} Nov 03 00:01:12.002038 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:12.002000 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bee1b-5899c6f99f-8ftj2" event={"ID":"940cf7dc-acaa-4ff3-b473-99c7536913cf","Type":"ContainerStarted","Data":"62bc16fb099378296e5eddf5e3c6c3204139be052311df3db87e72ac411bde1d"} Nov 03 00:01:12.002559 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:12.002296 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bee1b-5899c6f99f-8ftj2" Nov 03 00:01:12.003708 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:12.003684 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bee1b-64dd49579f-7psbr" event={"ID":"d6dcd3f9-d170-404b-9921-83b0d2e47ed4","Type":"ContainerStarted","Data":"a891654076a884a2a8fc9804fdcef701771653a431667caea89def289c9d63ee"} Nov 03 00:01:12.003852 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:12.003727 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bee1b-64dd49579f-7psbr" event={"ID":"d6dcd3f9-d170-404b-9921-83b0d2e47ed4","Type":"ContainerStarted","Data":"34f3add97d1a94e4eb2df68d23bca4f2c38951ba74d8b81a46c74fc6ab24b013"} Nov 03 00:01:12.003852 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:12.003759 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bee1b-64dd49579f-7psbr" Nov 03 00:01:12.117771 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:12.117718 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bee1b-5899c6f99f-8ftj2" podStartSLOduration=2.117704361 podStartE2EDuration="2.117704361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:01:12.115784478 +0000 UTC m=+4358.354282140" watchObservedRunningTime="2026-11-03 00:01:12.117704361 +0000 UTC m=+4358.356202022" Nov 03 00:01:12.231978 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:12.231731 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bee1b-64dd49579f-7psbr" podStartSLOduration=2.2317137750000002 podStartE2EDuration="2.231713775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:01:12.231119424 +0000 UTC m=+4358.469617109" watchObservedRunningTime="2026-11-03 00:01:12.231713775 +0000 UTC m=+4358.470211438" Nov 03 00:01:23.017009 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:23.016979 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bee1b-5899c6f99f-8ftj2" Nov 03 00:01:23.017492 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:01:23.017051 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bee1b-64dd49579f-7psbr" Nov 03 00:01:27.847355 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:01:27.847313 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:01:57.846542 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:01:57.846505 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:02:27.846908 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:02:27.846873 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:02:43.617186 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:02:43.614748 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-04713b77-7d67646d57-w65qh"] Nov 03 00:02:43.620989 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:02:43.620963 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-04713b77-7d67646d57-w65qh" Nov 03 00:02:43.672560 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:02:43.672426 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-04713b77-7d67646d57-w65qh"] Nov 03 00:02:43.674019 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:02:43.673998 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-04713b77-dockercfg-bzgnd\"" Nov 03 00:02:43.716234 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:02:43.716199 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-04713b77\"" Nov 03 00:02:43.734811 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:02:43.734783 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d587d22d-31c7-4266-a7df-323f11fd4499-plugins-config-volume\") pod \"payload-processing-e2e-mt-04713b77-7d67646d57-w65qh\" (UID: \"d587d22d-31c7-4266-a7df-323f11fd4499\") " pod="openshift-ingress/payload-processing-e2e-mt-04713b77-7d67646d57-w65qh" Nov 03 00:02:43.734941 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:02:43.734915 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fz6w\" (UniqueName: \"kubernetes.io/projected/d587d22d-31c7-4266-a7df-323f11fd4499-kube-api-access-9fz6w\") pod \"payload-processing-e2e-mt-04713b77-7d67646d57-w65qh\" (UID: \"d587d22d-31c7-4266-a7df-323f11fd4499\") " pod="openshift-ingress/payload-processing-e2e-mt-04713b77-7d67646d57-w65qh" Nov 03 00:02:43.835764 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:02:43.835731 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fz6w\" (UniqueName: \"kubernetes.io/projected/d587d22d-31c7-4266-a7df-323f11fd4499-kube-api-access-9fz6w\") pod \"payload-processing-e2e-mt-04713b77-7d67646d57-w65qh\" (UID: \"d587d22d-31c7-4266-a7df-323f11fd4499\") " pod="openshift-ingress/payload-processing-e2e-mt-04713b77-7d67646d57-w65qh" Nov 03 00:02:43.835953 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:02:43.835910 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d587d22d-31c7-4266-a7df-323f11fd4499-plugins-config-volume\") pod \"payload-processing-e2e-mt-04713b77-7d67646d57-w65qh\" (UID: \"d587d22d-31c7-4266-a7df-323f11fd4499\") " pod="openshift-ingress/payload-processing-e2e-mt-04713b77-7d67646d57-w65qh" Nov 03 00:02:43.836635 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:02:43.836613 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d587d22d-31c7-4266-a7df-323f11fd4499-plugins-config-volume\") pod \"payload-processing-e2e-mt-04713b77-7d67646d57-w65qh\" (UID: \"d587d22d-31c7-4266-a7df-323f11fd4499\") " pod="openshift-ingress/payload-processing-e2e-mt-04713b77-7d67646d57-w65qh" Nov 03 00:02:43.896734 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:02:43.896665 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fz6w\" (UniqueName: \"kubernetes.io/projected/d587d22d-31c7-4266-a7df-323f11fd4499-kube-api-access-9fz6w\") pod \"payload-processing-e2e-mt-04713b77-7d67646d57-w65qh\" (UID: \"d587d22d-31c7-4266-a7df-323f11fd4499\") " pod="openshift-ingress/payload-processing-e2e-mt-04713b77-7d67646d57-w65qh" Nov 03 00:02:43.944456 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:02:43.944416 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-04713b77-7d67646d57-w65qh" Nov 03 00:02:44.333077 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:02:44.333047 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-04713b77-7d67646d57-w65qh"] Nov 03 00:02:44.334053 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:02:44.334030 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd587d22d_31c7_4266_a7df_323f11fd4499.slice/crio-909d1f63f951c76e8f9e504cdc5d7c3de6a3d9d27394f9775131c74462b9efd3 WatchSource:0}: Error finding container 909d1f63f951c76e8f9e504cdc5d7c3de6a3d9d27394f9775131c74462b9efd3: Status 404 returned error can't find the container with id 909d1f63f951c76e8f9e504cdc5d7c3de6a3d9d27394f9775131c74462b9efd3 Nov 03 00:02:44.727868 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:02:44.727774 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-04713b77-7d67646d57-w65qh" event={"ID":"d587d22d-31c7-4266-a7df-323f11fd4499","Type":"ContainerStarted","Data":"276fb87e798d953642102aceba184530820981ba7d5fd09c5a6e11bd22bfbee6"} Nov 03 00:02:44.727868 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:02:44.727842 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-04713b77-7d67646d57-w65qh" event={"ID":"d587d22d-31c7-4266-a7df-323f11fd4499","Type":"ContainerStarted","Data":"909d1f63f951c76e8f9e504cdc5d7c3de6a3d9d27394f9775131c74462b9efd3"} Nov 03 00:02:44.728399 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:02:44.727978 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-04713b77-7d67646d57-w65qh" Nov 03 00:02:44.842892 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:02:44.842834 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-04713b77-7d67646d57-w65qh" podStartSLOduration=1.842819465 podStartE2EDuration="1.842819465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:02:44.842142344 +0000 UTC m=+4451.080640031" watchObservedRunningTime="2026-11-03 00:02:44.842819465 +0000 UTC m=+4451.081317128" Nov 03 00:02:55.738228 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:02:55.738198 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-04713b77-7d67646d57-w65qh" Nov 03 00:02:57.848787 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:02:57.848750 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:03:27.846173 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:03:27.846093 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:03:34.659968 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:34.659932 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-440c60-65b68b4cd7-82p9f"] Nov 03 00:03:34.904372 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:34.904339 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 03 00:03:34.922358 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:34.922289 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 03 00:03:35.013886 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:35.013857 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 03 00:03:35.031698 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:35.031667 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 03 00:03:39.660437 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:39.660392 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-440c60-65b68b4cd7-82p9f" podUID="1d559e50-9c39-42dd-8ac1-6d42ebb726c4" containerName="payload-processing" containerID="cri-o://5f0c6fee4d188c1fc478716bb4842808741dbe8d1145bdb8d9fbb3a12d4d6b48" gracePeriod=25 Nov 03 00:03:41.450111 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:41.450061 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-440c60-65b68b4cd7-82p9f" podUID="1d559e50-9c39-42dd-8ac1-6d42ebb726c4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:03:45.304666 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:45.304586 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f46b61-9bf775457-hdjr6"] Nov 03 00:03:48.853858 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:48.853814 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz"] Nov 03 00:03:48.854334 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:48.854103 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz" podUID="30badf9d-6a37-4a2a-ab5d-13c25bbec12c" containerName="maas-api" containerID="cri-o://fdb0f47bb8372ca5c0459cf5c89e4668eccb48b04482c7dded8c39a119050b8b" gracePeriod=30 Nov 03 00:03:49.151369 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:49.151338 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz" Nov 03 00:03:49.230471 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:49.230440 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30badf9d-6a37-4a2a-ab5d-13c25bbec12c-kube-api-access-b9j8d\" (UniqueName: \"kubernetes.io/projected/30badf9d-6a37-4a2a-ab5d-13c25bbec12c-kube-api-access-b9j8d\") pod \"30badf9d-6a37-4a2a-ab5d-13c25bbec12c\" (UID: \"30badf9d-6a37-4a2a-ab5d-13c25bbec12c\") " Nov 03 00:03:49.230627 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:49.230513 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30badf9d-6a37-4a2a-ab5d-13c25bbec12c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30badf9d-6a37-4a2a-ab5d-13c25bbec12c-maas-api-tls\") pod \"30badf9d-6a37-4a2a-ab5d-13c25bbec12c\" (UID: \"30badf9d-6a37-4a2a-ab5d-13c25bbec12c\") " Nov 03 00:03:49.230627 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:49.230583 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30badf9d-6a37-4a2a-ab5d-13c25bbec12c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30badf9d-6a37-4a2a-ab5d-13c25bbec12c-openshift-service-ca\") pod \"30badf9d-6a37-4a2a-ab5d-13c25bbec12c\" (UID: \"30badf9d-6a37-4a2a-ab5d-13c25bbec12c\") " Nov 03 00:03:49.230627 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:49.230616 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30badf9d-6a37-4a2a-ab5d-13c25bbec12c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/30badf9d-6a37-4a2a-ab5d-13c25bbec12c-metrics-certs\") pod \"30badf9d-6a37-4a2a-ab5d-13c25bbec12c\" (UID: \"30badf9d-6a37-4a2a-ab5d-13c25bbec12c\") " Nov 03 00:03:49.230988 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:49.230951 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30badf9d-6a37-4a2a-ab5d-13c25bbec12c-openshift-service-ca" pod "30badf9d-6a37-4a2a-ab5d-13c25bbec12c" (UID: "30badf9d-6a37-4a2a-ab5d-13c25bbec12c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:03:49.231221 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:49.231203 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30badf9d-6a37-4a2a-ab5d-13c25bbec12c-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:03:49.232874 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:49.232846 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30badf9d-6a37-4a2a-ab5d-13c25bbec12c-maas-api-tls" pod "30badf9d-6a37-4a2a-ab5d-13c25bbec12c" (UID: "30badf9d-6a37-4a2a-ab5d-13c25bbec12c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:03:49.232990 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:49.232916 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30badf9d-6a37-4a2a-ab5d-13c25bbec12c-kube-api-access-b9j8d" pod "30badf9d-6a37-4a2a-ab5d-13c25bbec12c" (UID: "30badf9d-6a37-4a2a-ab5d-13c25bbec12c"). InnerVolumeSpecName "kube-api-access-b9j8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:03:49.232990 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:49.232974 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30badf9d-6a37-4a2a-ab5d-13c25bbec12c-metrics-certs" pod "30badf9d-6a37-4a2a-ab5d-13c25bbec12c" (UID: "30badf9d-6a37-4a2a-ab5d-13c25bbec12c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:03:49.253834 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:49.253808 5727 generic.go:356] "Generic (PLEG): container finished" podID="30badf9d-6a37-4a2a-ab5d-13c25bbec12c" containerID="fdb0f47bb8372ca5c0459cf5c89e4668eccb48b04482c7dded8c39a119050b8b" exitCode=0 Nov 03 00:03:49.253927 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:49.253856 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz" Nov 03 00:03:49.253927 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:49.253895 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz" event={"ID":"30badf9d-6a37-4a2a-ab5d-13c25bbec12c","Type":"ContainerDied","Data":"fdb0f47bb8372ca5c0459cf5c89e4668eccb48b04482c7dded8c39a119050b8b"} Nov 03 00:03:49.254001 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:49.253961 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz" event={"ID":"30badf9d-6a37-4a2a-ab5d-13c25bbec12c","Type":"ContainerDied","Data":"3b8bff805d66f5e8608b6ea5844e52ec234f22601ddcd4ed9b0a65568e7d3f7a"} Nov 03 00:03:49.254001 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:49.253993 5727 scope.go:122] "RemoveContainer" containerID="fdb0f47bb8372ca5c0459cf5c89e4668eccb48b04482c7dded8c39a119050b8b" Nov 03 00:03:49.266773 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:49.266752 5727 scope.go:122] "RemoveContainer" containerID="fdb0f47bb8372ca5c0459cf5c89e4668eccb48b04482c7dded8c39a119050b8b" Nov 03 00:03:49.267536 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:03:49.267415 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdb0f47bb8372ca5c0459cf5c89e4668eccb48b04482c7dded8c39a119050b8b\": container with ID starting with fdb0f47bb8372ca5c0459cf5c89e4668eccb48b04482c7dded8c39a119050b8b not found: ID does not exist" containerID="fdb0f47bb8372ca5c0459cf5c89e4668eccb48b04482c7dded8c39a119050b8b" Nov 03 00:03:49.267536 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:49.267492 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdb0f47bb8372ca5c0459cf5c89e4668eccb48b04482c7dded8c39a119050b8b"} err="failed to get container status \"fdb0f47bb8372ca5c0459cf5c89e4668eccb48b04482c7dded8c39a119050b8b\": rpc error: code = NotFound desc = could not find container \"fdb0f47bb8372ca5c0459cf5c89e4668eccb48b04482c7dded8c39a119050b8b\": container with ID starting with fdb0f47bb8372ca5c0459cf5c89e4668eccb48b04482c7dded8c39a119050b8b not found: ID does not exist" Nov 03 00:03:49.337287 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:49.337232 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30badf9d-6a37-4a2a-ab5d-13c25bbec12c-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:03:49.337618 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:49.337597 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30badf9d-6a37-4a2a-ab5d-13c25bbec12c-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:03:49.337786 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:49.337767 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9j8d\" (UniqueName: \"kubernetes.io/projected/30badf9d-6a37-4a2a-ab5d-13c25bbec12c-kube-api-access-b9j8d\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:03:49.379390 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:49.379319 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz"] Nov 03 00:03:49.384991 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:49.384967 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-440c60-6fb556c87f-tj9gz"] Nov 03 00:03:49.925318 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:49.925283 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30badf9d-6a37-4a2a-ab5d-13c25bbec12c" path="/var/lib/kubelet/pods/30badf9d-6a37-4a2a-ab5d-13c25bbec12c/volumes" Nov 03 00:03:50.305113 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.305019 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f46b61-9bf775457-hdjr6" podUID="4b213bcf-90cf-414e-9432-e7887fa901fa" containerName="payload-processing" containerID="cri-o://11be3f531df24d6c9014125f931cfd4fd7699bcbcec4b553bc78abd2e28ecd5b" gracePeriod=25 Nov 03 00:03:50.341106 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.341074 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9b6c9e-6d4bc8b6f9-dztdx"] Nov 03 00:03:50.341752 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.341739 5727 cpu_manager.go:414] "containerMap: removing container" podUID="30badf9d-6a37-4a2a-ab5d-13c25bbec12c" containerName="maas-api" Nov 03 00:03:50.341799 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.341754 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30badf9d-6a37-4a2a-ab5d-13c25bbec12c" containerName="maas-api" Nov 03 00:03:50.341857 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.341849 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="30badf9d-6a37-4a2a-ab5d-13c25bbec12c" containerName="maas-api" Nov 03 00:03:50.346427 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.346412 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9b6c9e-6d4bc8b6f9-dztdx" Nov 03 00:03:50.398825 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.398799 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-9b6c9e\"" Nov 03 00:03:50.462158 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.450843 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-9b6c9e-dockercfg-7pw2c\"" Nov 03 00:03:50.462158 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.455008 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9b6c9e-6d4bc8b6f9-dztdx"] Nov 03 00:03:50.462158 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.455952 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9cht\" (UniqueName: \"kubernetes.io/projected/4bd218d6-1000-4fce-bade-c54fa8e7026b-kube-api-access-f9cht\") pod \"payload-pre-processing-e2e-shared-b-w1-9b6c9e-6d4bc8b6f9-dztdx\" (UID: \"4bd218d6-1000-4fce-bade-c54fa8e7026b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9b6c9e-6d4bc8b6f9-dztdx" Nov 03 00:03:50.462158 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.456107 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd218d6-1000-4fce-bade-c54fa8e7026b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-9b6c9e-6d4bc8b6f9-dztdx\" (UID: \"4bd218d6-1000-4fce-bade-c54fa8e7026b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9b6c9e-6d4bc8b6f9-dztdx" Nov 03 00:03:50.494800 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.494764 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9b6c9e-54cc6f6cdf-dr7kw"] Nov 03 00:03:50.498948 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.498925 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9b6c9e-54cc6f6cdf-dr7kw" Nov 03 00:03:50.554237 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.554204 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9b6c9e-54cc6f6cdf-dr7kw"] Nov 03 00:03:50.557363 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.557296 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49421404-0145-4fda-a9a4-26c7ba952374-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-9b6c9e-54cc6f6cdf-dr7kw\" (UID: \"49421404-0145-4fda-a9a4-26c7ba952374\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9b6c9e-54cc6f6cdf-dr7kw" Nov 03 00:03:50.557363 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.557350 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd218d6-1000-4fce-bade-c54fa8e7026b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-9b6c9e-6d4bc8b6f9-dztdx\" (UID: \"4bd218d6-1000-4fce-bade-c54fa8e7026b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9b6c9e-6d4bc8b6f9-dztdx" Nov 03 00:03:50.557537 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.557441 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq47q\" (UniqueName: \"kubernetes.io/projected/49421404-0145-4fda-a9a4-26c7ba952374-kube-api-access-vq47q\") pod \"payload-processing-e2e-shared-b-w1-9b6c9e-54cc6f6cdf-dr7kw\" (UID: \"49421404-0145-4fda-a9a4-26c7ba952374\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9b6c9e-54cc6f6cdf-dr7kw" Nov 03 00:03:50.557605 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.557583 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9cht\" (UniqueName: \"kubernetes.io/projected/4bd218d6-1000-4fce-bade-c54fa8e7026b-kube-api-access-f9cht\") pod \"payload-pre-processing-e2e-shared-b-w1-9b6c9e-6d4bc8b6f9-dztdx\" (UID: \"4bd218d6-1000-4fce-bade-c54fa8e7026b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9b6c9e-6d4bc8b6f9-dztdx" Nov 03 00:03:50.558275 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.558251 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd218d6-1000-4fce-bade-c54fa8e7026b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-9b6c9e-6d4bc8b6f9-dztdx\" (UID: \"4bd218d6-1000-4fce-bade-c54fa8e7026b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9b6c9e-6d4bc8b6f9-dztdx" Nov 03 00:03:50.621451 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.621425 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9cht\" (UniqueName: \"kubernetes.io/projected/4bd218d6-1000-4fce-bade-c54fa8e7026b-kube-api-access-f9cht\") pod \"payload-pre-processing-e2e-shared-b-w1-9b6c9e-6d4bc8b6f9-dztdx\" (UID: \"4bd218d6-1000-4fce-bade-c54fa8e7026b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9b6c9e-6d4bc8b6f9-dztdx" Nov 03 00:03:50.658775 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.658745 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49421404-0145-4fda-a9a4-26c7ba952374-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-9b6c9e-54cc6f6cdf-dr7kw\" (UID: \"49421404-0145-4fda-a9a4-26c7ba952374\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9b6c9e-54cc6f6cdf-dr7kw" Nov 03 00:03:50.658928 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.658800 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq47q\" (UniqueName: \"kubernetes.io/projected/49421404-0145-4fda-a9a4-26c7ba952374-kube-api-access-vq47q\") pod \"payload-processing-e2e-shared-b-w1-9b6c9e-54cc6f6cdf-dr7kw\" (UID: \"49421404-0145-4fda-a9a4-26c7ba952374\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9b6c9e-54cc6f6cdf-dr7kw" Nov 03 00:03:50.659474 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.659455 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49421404-0145-4fda-a9a4-26c7ba952374-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-9b6c9e-54cc6f6cdf-dr7kw\" (UID: \"49421404-0145-4fda-a9a4-26c7ba952374\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9b6c9e-54cc6f6cdf-dr7kw" Nov 03 00:03:50.660496 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.660473 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9b6c9e-6d4bc8b6f9-dztdx" Nov 03 00:03:50.667291 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.667269 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr"] Nov 03 00:03:50.676392 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.676363 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr" Nov 03 00:03:50.719953 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.719916 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq47q\" (UniqueName: \"kubernetes.io/projected/49421404-0145-4fda-a9a4-26c7ba952374-kube-api-access-vq47q\") pod \"payload-processing-e2e-shared-b-w1-9b6c9e-54cc6f6cdf-dr7kw\" (UID: \"49421404-0145-4fda-a9a4-26c7ba952374\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9b6c9e-54cc6f6cdf-dr7kw" Nov 03 00:03:50.740525 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.733546 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr"] Nov 03 00:03:50.760375 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.760334 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b693031-94ce-4930-b763-6455d23bfea8-maas-api-tls\") pod \"maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr\" (UID: \"3b693031-94ce-4930-b763-6455d23bfea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr" Nov 03 00:03:50.760550 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.760395 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b693031-94ce-4930-b763-6455d23bfea8-openshift-service-ca\") pod \"maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr\" (UID: \"3b693031-94ce-4930-b763-6455d23bfea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr" Nov 03 00:03:50.760550 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.760454 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kgjw\" (UniqueName: \"kubernetes.io/projected/3b693031-94ce-4930-b763-6455d23bfea8-kube-api-access-4kgjw\") pod \"maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr\" (UID: \"3b693031-94ce-4930-b763-6455d23bfea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr" Nov 03 00:03:50.760689 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.760558 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b693031-94ce-4930-b763-6455d23bfea8-metrics-certs\") pod \"maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr\" (UID: \"3b693031-94ce-4930-b763-6455d23bfea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr" Nov 03 00:03:50.776835 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.776808 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-04713b77-serving-cert\"" Nov 03 00:03:50.813839 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.813814 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9b6c9e-54cc6f6cdf-dr7kw" Nov 03 00:03:50.861386 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.861352 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kgjw\" (UniqueName: \"kubernetes.io/projected/3b693031-94ce-4930-b763-6455d23bfea8-kube-api-access-4kgjw\") pod \"maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr\" (UID: \"3b693031-94ce-4930-b763-6455d23bfea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr" Nov 03 00:03:50.861545 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.861468 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b693031-94ce-4930-b763-6455d23bfea8-metrics-certs\") pod \"maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr\" (UID: \"3b693031-94ce-4930-b763-6455d23bfea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr" Nov 03 00:03:50.861545 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.861506 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b693031-94ce-4930-b763-6455d23bfea8-maas-api-tls\") pod \"maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr\" (UID: \"3b693031-94ce-4930-b763-6455d23bfea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr" Nov 03 00:03:50.861545 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.861540 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b693031-94ce-4930-b763-6455d23bfea8-openshift-service-ca\") pod \"maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr\" (UID: \"3b693031-94ce-4930-b763-6455d23bfea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr" Nov 03 00:03:50.861827 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:03:50.861804 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-04713b77-serving-cert: secret "maas-api-e2e-mt-04713b77-serving-cert" not found Nov 03 00:03:50.861914 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:03:50.861872 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b693031-94ce-4930-b763-6455d23bfea8-maas-api-tls podName:3b693031-94ce-4930-b763-6455d23bfea8 nodeName:}" failed. No retries permitted until 2026-11-03 00:03:51.361851981 +0000 UTC m=+4517.600349642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3b693031-94ce-4930-b763-6455d23bfea8-maas-api-tls") pod "maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr" (UID: "3b693031-94ce-4930-b763-6455d23bfea8") : secret "maas-api-e2e-mt-04713b77-serving-cert" not found Nov 03 00:03:50.862122 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.862100 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b693031-94ce-4930-b763-6455d23bfea8-openshift-service-ca\") pod \"maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr\" (UID: \"3b693031-94ce-4930-b763-6455d23bfea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr" Nov 03 00:03:50.864503 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.864479 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b693031-94ce-4930-b763-6455d23bfea8-metrics-certs\") pod \"maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr\" (UID: \"3b693031-94ce-4930-b763-6455d23bfea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr" Nov 03 00:03:50.925434 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:50.924253 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kgjw\" (UniqueName: \"kubernetes.io/projected/3b693031-94ce-4930-b763-6455d23bfea8-kube-api-access-4kgjw\") pod \"maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr\" (UID: \"3b693031-94ce-4930-b763-6455d23bfea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr" Nov 03 00:03:51.056557 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:51.056528 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9b6c9e-6d4bc8b6f9-dztdx"] Nov 03 00:03:51.057438 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:03:51.057411 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd218d6_1000_4fce_bade_c54fa8e7026b.slice/crio-2b9eb6441452397850a52c032aa121c73182a7aba948a409c48fe0f124a9fd89 WatchSource:0}: Error finding container 2b9eb6441452397850a52c032aa121c73182a7aba948a409c48fe0f124a9fd89: Status 404 returned error can't find the container with id 2b9eb6441452397850a52c032aa121c73182a7aba948a409c48fe0f124a9fd89 Nov 03 00:03:51.196302 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:51.196274 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9b6c9e-54cc6f6cdf-dr7kw"] Nov 03 00:03:51.197344 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:03:51.197319 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49421404_0145_4fda_a9a4_26c7ba952374.slice/crio-aca9ee27946491e91e1babdd672aed621162ec5a2615346a62ff5c9f536cf0bc WatchSource:0}: Error finding container aca9ee27946491e91e1babdd672aed621162ec5a2615346a62ff5c9f536cf0bc: Status 404 returned error can't find the container with id aca9ee27946491e91e1babdd672aed621162ec5a2615346a62ff5c9f536cf0bc Nov 03 00:03:51.274682 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:51.274654 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9b6c9e-6d4bc8b6f9-dztdx" event={"ID":"4bd218d6-1000-4fce-bade-c54fa8e7026b","Type":"ContainerStarted","Data":"981c3089f1f5c517bfe95b024ea43a5d483029d67cfdea62393a918ae4bc4e1d"} Nov 03 00:03:51.274802 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:51.274714 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9b6c9e-6d4bc8b6f9-dztdx" event={"ID":"4bd218d6-1000-4fce-bade-c54fa8e7026b","Type":"ContainerStarted","Data":"2b9eb6441452397850a52c032aa121c73182a7aba948a409c48fe0f124a9fd89"} Nov 03 00:03:51.274891 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:51.274869 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9b6c9e-6d4bc8b6f9-dztdx" Nov 03 00:03:51.276549 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:51.276523 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9b6c9e-54cc6f6cdf-dr7kw" event={"ID":"49421404-0145-4fda-a9a4-26c7ba952374","Type":"ContainerStarted","Data":"cbf3cf4e32644d550ca821dd60c42ade5f93b3edc2f6cd3023cd093ff1efcb56"} Nov 03 00:03:51.276876 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:51.276575 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9b6c9e-54cc6f6cdf-dr7kw" Nov 03 00:03:51.276876 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:51.276599 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9b6c9e-54cc6f6cdf-dr7kw" event={"ID":"49421404-0145-4fda-a9a4-26c7ba952374","Type":"ContainerStarted","Data":"aca9ee27946491e91e1babdd672aed621162ec5a2615346a62ff5c9f536cf0bc"} Nov 03 00:03:51.368021 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:51.367947 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b693031-94ce-4930-b763-6455d23bfea8-maas-api-tls\") pod \"maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr\" (UID: \"3b693031-94ce-4930-b763-6455d23bfea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr" Nov 03 00:03:51.370452 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:51.370429 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b693031-94ce-4930-b763-6455d23bfea8-maas-api-tls\") pod \"maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr\" (UID: \"3b693031-94ce-4930-b763-6455d23bfea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr" Nov 03 00:03:51.389467 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:51.389420 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9b6c9e-6d4bc8b6f9-dztdx" podStartSLOduration=1.389407865 podStartE2EDuration="1.389407865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:03:51.3889565 +0000 UTC m=+4517.627454165" watchObservedRunningTime="2026-11-03 00:03:51.389407865 +0000 UTC m=+4517.627905847" Nov 03 00:03:51.449780 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:51.449750 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-440c60-65b68b4cd7-82p9f" podUID="1d559e50-9c39-42dd-8ac1-6d42ebb726c4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:03:51.508070 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:51.508001 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9b6c9e-54cc6f6cdf-dr7kw" podStartSLOduration=1.5079813770000001 podStartE2EDuration="1.507981377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:03:51.503929931 +0000 UTC m=+4517.742427594" watchObservedRunningTime="2026-11-03 00:03:51.507981377 +0000 UTC m=+4517.746479044" Nov 03 00:03:51.624375 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:51.624287 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr" Nov 03 00:03:52.017175 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:52.017144 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr"] Nov 03 00:03:52.017843 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:03:52.017819 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b693031_94ce_4930_b763_6455d23bfea8.slice/crio-c87d291b0a8bfacabd9e6ec2dae2d5875899bde3c52e51dfbb3e2d406a079b55 WatchSource:0}: Error finding container c87d291b0a8bfacabd9e6ec2dae2d5875899bde3c52e51dfbb3e2d406a079b55: Status 404 returned error can't find the container with id c87d291b0a8bfacabd9e6ec2dae2d5875899bde3c52e51dfbb3e2d406a079b55 Nov 03 00:03:52.019520 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:52.019493 5727 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:03:52.286244 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:52.286155 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr" event={"ID":"3b693031-94ce-4930-b763-6455d23bfea8","Type":"ContainerStarted","Data":"c87d291b0a8bfacabd9e6ec2dae2d5875899bde3c52e51dfbb3e2d406a079b55"} Nov 03 00:03:53.300514 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:53.298109 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr" event={"ID":"3b693031-94ce-4930-b763-6455d23bfea8","Type":"ContainerStarted","Data":"f7222ef1e8bae8fc15a535b083c5497f8b73bac888ffe57d0d8063c15ab45173"} Nov 03 00:03:53.300514 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:53.298204 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr" Nov 03 00:03:53.417578 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:53.417528 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr" podStartSLOduration=2.594371323 podStartE2EDuration="3.417513653s" totalImagesPullingTime="823.14233ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:03:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:03:53.413560426 +0000 UTC m=+4519.652058089" watchObservedRunningTime="2026-11-03 00:03:53.417513653 +0000 UTC m=+4519.656011346" Nov 03 00:03:54.016981 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:54.016943 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f46b61-9bf775457-hdjr6" podUID="4b213bcf-90cf-414e-9432-e7887fa901fa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:03:57.850413 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:03:57.850370 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:03:59.313233 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:59.313202 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr" Nov 03 00:03:59.942493 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:03:59.942463 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-440c60-65b68b4cd7-82p9f" Nov 03 00:04:00.094322 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:00.094287 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d559e50-9c39-42dd-8ac1-6d42ebb726c4-kube-api-access-cbsvb\" (UniqueName: \"kubernetes.io/projected/1d559e50-9c39-42dd-8ac1-6d42ebb726c4-kube-api-access-cbsvb\") pod \"1d559e50-9c39-42dd-8ac1-6d42ebb726c4\" (UID: \"1d559e50-9c39-42dd-8ac1-6d42ebb726c4\") " Nov 03 00:04:00.094513 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:00.094401 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d559e50-9c39-42dd-8ac1-6d42ebb726c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d559e50-9c39-42dd-8ac1-6d42ebb726c4-plugins-config-volume\") pod \"1d559e50-9c39-42dd-8ac1-6d42ebb726c4\" (UID: \"1d559e50-9c39-42dd-8ac1-6d42ebb726c4\") " Nov 03 00:04:00.097319 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:00.097285 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d559e50-9c39-42dd-8ac1-6d42ebb726c4-kube-api-access-cbsvb" pod "1d559e50-9c39-42dd-8ac1-6d42ebb726c4" (UID: "1d559e50-9c39-42dd-8ac1-6d42ebb726c4"). InnerVolumeSpecName "kube-api-access-cbsvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:04:00.098368 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:00.098335 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d559e50-9c39-42dd-8ac1-6d42ebb726c4-plugins-config-volume" pod "1d559e50-9c39-42dd-8ac1-6d42ebb726c4" (UID: "1d559e50-9c39-42dd-8ac1-6d42ebb726c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:04:00.196368 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:00.196300 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbsvb\" (UniqueName: \"kubernetes.io/projected/1d559e50-9c39-42dd-8ac1-6d42ebb726c4-kube-api-access-cbsvb\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:04:00.196368 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:00.196334 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d559e50-9c39-42dd-8ac1-6d42ebb726c4-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:04:00.356113 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:00.356081 5727 generic.go:356] "Generic (PLEG): container finished" podID="1d559e50-9c39-42dd-8ac1-6d42ebb726c4" containerID="5f0c6fee4d188c1fc478716bb4842808741dbe8d1145bdb8d9fbb3a12d4d6b48" exitCode=0 Nov 03 00:04:00.356528 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:00.356167 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-440c60-65b68b4cd7-82p9f" Nov 03 00:04:00.356528 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:00.356168 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-440c60-65b68b4cd7-82p9f" event={"ID":"1d559e50-9c39-42dd-8ac1-6d42ebb726c4","Type":"ContainerDied","Data":"5f0c6fee4d188c1fc478716bb4842808741dbe8d1145bdb8d9fbb3a12d4d6b48"} Nov 03 00:04:00.356528 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:00.356230 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-440c60-65b68b4cd7-82p9f" event={"ID":"1d559e50-9c39-42dd-8ac1-6d42ebb726c4","Type":"ContainerDied","Data":"b8e2e3031d74be5e28b75357c96a10d1f0501305123888b41e4a7ebf54815c1b"} Nov 03 00:04:00.356528 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:00.356262 5727 scope.go:122] "RemoveContainer" containerID="5f0c6fee4d188c1fc478716bb4842808741dbe8d1145bdb8d9fbb3a12d4d6b48" Nov 03 00:04:00.366939 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:00.366921 5727 scope.go:122] "RemoveContainer" containerID="5f0c6fee4d188c1fc478716bb4842808741dbe8d1145bdb8d9fbb3a12d4d6b48" Nov 03 00:04:00.367238 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:04:00.367219 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f0c6fee4d188c1fc478716bb4842808741dbe8d1145bdb8d9fbb3a12d4d6b48\": container with ID starting with 5f0c6fee4d188c1fc478716bb4842808741dbe8d1145bdb8d9fbb3a12d4d6b48 not found: ID does not exist" containerID="5f0c6fee4d188c1fc478716bb4842808741dbe8d1145bdb8d9fbb3a12d4d6b48" Nov 03 00:04:00.367293 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:00.367248 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f0c6fee4d188c1fc478716bb4842808741dbe8d1145bdb8d9fbb3a12d4d6b48"} err="failed to get container status \"5f0c6fee4d188c1fc478716bb4842808741dbe8d1145bdb8d9fbb3a12d4d6b48\": rpc error: code = NotFound desc = could not find container \"5f0c6fee4d188c1fc478716bb4842808741dbe8d1145bdb8d9fbb3a12d4d6b48\": container with ID starting with 5f0c6fee4d188c1fc478716bb4842808741dbe8d1145bdb8d9fbb3a12d4d6b48 not found: ID does not exist" Nov 03 00:04:00.520425 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:00.520281 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-440c60-65b68b4cd7-82p9f"] Nov 03 00:04:00.569827 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:00.569791 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-440c60-65b68b4cd7-82p9f"] Nov 03 00:04:01.887811 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:01.887772 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d559e50-9c39-42dd-8ac1-6d42ebb726c4" path="/var/lib/kubelet/pods/1d559e50-9c39-42dd-8ac1-6d42ebb726c4/volumes" Nov 03 00:04:02.289167 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:02.289053 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9b6c9e-54cc6f6cdf-dr7kw" Nov 03 00:04:02.289167 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:02.289155 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9b6c9e-6d4bc8b6f9-dztdx" Nov 03 00:04:03.810732 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:03.810700 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-915ccb-57c5454f5c-ggx2n"] Nov 03 00:04:04.017222 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:04.017164 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f46b61-9bf775457-hdjr6" podUID="4b213bcf-90cf-414e-9432-e7887fa901fa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:04:08.811438 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:08.811374 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-915ccb-57c5454f5c-ggx2n" podUID="0a89e95d-54df-4506-94ec-34f333911ef7" containerName="payload-processing" containerID="cri-o://f3bc723ae3ee5c85cd2b81500071bc2c904ca453f4e15ac7412b064018301ca3" gracePeriod=25 Nov 03 00:04:10.446277 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:10.445998 5727 generic.go:356] "Generic (PLEG): container finished" podID="4b213bcf-90cf-414e-9432-e7887fa901fa" containerID="11be3f531df24d6c9014125f931cfd4fd7699bcbcec4b553bc78abd2e28ecd5b" exitCode=0 Nov 03 00:04:10.446277 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:10.446162 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f46b61-9bf775457-hdjr6" event={"ID":"4b213bcf-90cf-414e-9432-e7887fa901fa","Type":"ContainerDied","Data":"11be3f531df24d6c9014125f931cfd4fd7699bcbcec4b553bc78abd2e28ecd5b"} Nov 03 00:04:10.588914 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:10.588888 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f46b61-9bf775457-hdjr6" Nov 03 00:04:10.636919 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:10.636889 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b213bcf-90cf-414e-9432-e7887fa901fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b213bcf-90cf-414e-9432-e7887fa901fa-plugins-config-volume\") pod \"4b213bcf-90cf-414e-9432-e7887fa901fa\" (UID: \"4b213bcf-90cf-414e-9432-e7887fa901fa\") " Nov 03 00:04:10.637088 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:10.637005 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b213bcf-90cf-414e-9432-e7887fa901fa-kube-api-access-wc2cc\" (UniqueName: \"kubernetes.io/projected/4b213bcf-90cf-414e-9432-e7887fa901fa-kube-api-access-wc2cc\") pod \"4b213bcf-90cf-414e-9432-e7887fa901fa\" (UID: \"4b213bcf-90cf-414e-9432-e7887fa901fa\") " Nov 03 00:04:10.637289 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:10.637255 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b213bcf-90cf-414e-9432-e7887fa901fa-plugins-config-volume" pod "4b213bcf-90cf-414e-9432-e7887fa901fa" (UID: "4b213bcf-90cf-414e-9432-e7887fa901fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:04:10.637434 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:10.637410 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b213bcf-90cf-414e-9432-e7887fa901fa-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:04:10.639210 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:10.639188 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b213bcf-90cf-414e-9432-e7887fa901fa-kube-api-access-wc2cc" pod "4b213bcf-90cf-414e-9432-e7887fa901fa" (UID: "4b213bcf-90cf-414e-9432-e7887fa901fa"). InnerVolumeSpecName "kube-api-access-wc2cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:04:10.738740 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:10.738707 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc2cc\" (UniqueName: \"kubernetes.io/projected/4b213bcf-90cf-414e-9432-e7887fa901fa-kube-api-access-wc2cc\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:04:11.455025 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:11.454995 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f46b61-9bf775457-hdjr6" Nov 03 00:04:11.455487 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:11.455002 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f46b61-9bf775457-hdjr6" event={"ID":"4b213bcf-90cf-414e-9432-e7887fa901fa","Type":"ContainerDied","Data":"a4bc5fd02895a8aaa8dd5ec34aa5f459fb54fa1130966edb417bb511b205e2b6"} Nov 03 00:04:11.455487 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:11.455121 5727 scope.go:122] "RemoveContainer" containerID="11be3f531df24d6c9014125f931cfd4fd7699bcbcec4b553bc78abd2e28ecd5b" Nov 03 00:04:11.628469 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:11.628441 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f46b61-9bf775457-hdjr6"] Nov 03 00:04:11.677644 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:11.677618 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f46b61-9bf775457-hdjr6"] Nov 03 00:04:11.884216 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:11.884185 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b213bcf-90cf-414e-9432-e7887fa901fa" path="/var/lib/kubelet/pods/4b213bcf-90cf-414e-9432-e7887fa901fa/volumes" Nov 03 00:04:18.614480 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:18.614438 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-915ccb-57c5454f5c-ggx2n" podUID="0a89e95d-54df-4506-94ec-34f333911ef7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:04:27.847072 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:04:27.847030 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:04:28.613754 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:28.613718 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-915ccb-57c5454f5c-ggx2n" podUID="0a89e95d-54df-4506-94ec-34f333911ef7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:04:29.102201 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:29.102176 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-915ccb-57c5454f5c-ggx2n" Nov 03 00:04:29.256667 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:29.256569 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a89e95d-54df-4506-94ec-34f333911ef7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a89e95d-54df-4506-94ec-34f333911ef7-plugins-config-volume\") pod \"0a89e95d-54df-4506-94ec-34f333911ef7\" (UID: \"0a89e95d-54df-4506-94ec-34f333911ef7\") " Nov 03 00:04:29.256838 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:29.256704 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a89e95d-54df-4506-94ec-34f333911ef7-kube-api-access-56crt\" (UniqueName: \"kubernetes.io/projected/0a89e95d-54df-4506-94ec-34f333911ef7-kube-api-access-56crt\") pod \"0a89e95d-54df-4506-94ec-34f333911ef7\" (UID: \"0a89e95d-54df-4506-94ec-34f333911ef7\") " Nov 03 00:04:29.256987 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:29.256961 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a89e95d-54df-4506-94ec-34f333911ef7-plugins-config-volume" pod "0a89e95d-54df-4506-94ec-34f333911ef7" (UID: "0a89e95d-54df-4506-94ec-34f333911ef7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:04:29.257380 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:29.257358 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a89e95d-54df-4506-94ec-34f333911ef7-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:04:29.258928 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:29.258893 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a89e95d-54df-4506-94ec-34f333911ef7-kube-api-access-56crt" pod "0a89e95d-54df-4506-94ec-34f333911ef7" (UID: "0a89e95d-54df-4506-94ec-34f333911ef7"). InnerVolumeSpecName "kube-api-access-56crt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:04:29.358388 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:29.358362 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56crt\" (UniqueName: \"kubernetes.io/projected/0a89e95d-54df-4506-94ec-34f333911ef7-kube-api-access-56crt\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:04:29.594419 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:29.594384 5727 generic.go:356] "Generic (PLEG): container finished" podID="0a89e95d-54df-4506-94ec-34f333911ef7" containerID="f3bc723ae3ee5c85cd2b81500071bc2c904ca453f4e15ac7412b064018301ca3" exitCode=0 Nov 03 00:04:29.594606 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:29.594430 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-915ccb-57c5454f5c-ggx2n" Nov 03 00:04:29.594606 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:29.594471 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-915ccb-57c5454f5c-ggx2n" event={"ID":"0a89e95d-54df-4506-94ec-34f333911ef7","Type":"ContainerDied","Data":"f3bc723ae3ee5c85cd2b81500071bc2c904ca453f4e15ac7412b064018301ca3"} Nov 03 00:04:29.594606 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:29.594529 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-915ccb-57c5454f5c-ggx2n" event={"ID":"0a89e95d-54df-4506-94ec-34f333911ef7","Type":"ContainerDied","Data":"2d3abd952ca1a79e81aa3a859889c3d10c44a098a1f772c33c9874d36d40bcf3"} Nov 03 00:04:29.594606 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:29.594558 5727 scope.go:122] "RemoveContainer" containerID="f3bc723ae3ee5c85cd2b81500071bc2c904ca453f4e15ac7412b064018301ca3" Nov 03 00:04:29.605609 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:29.605588 5727 scope.go:122] "RemoveContainer" containerID="f3bc723ae3ee5c85cd2b81500071bc2c904ca453f4e15ac7412b064018301ca3" Nov 03 00:04:29.605872 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:04:29.605850 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3bc723ae3ee5c85cd2b81500071bc2c904ca453f4e15ac7412b064018301ca3\": container with ID starting with f3bc723ae3ee5c85cd2b81500071bc2c904ca453f4e15ac7412b064018301ca3 not found: ID does not exist" containerID="f3bc723ae3ee5c85cd2b81500071bc2c904ca453f4e15ac7412b064018301ca3" Nov 03 00:04:29.605919 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:29.605881 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3bc723ae3ee5c85cd2b81500071bc2c904ca453f4e15ac7412b064018301ca3"} err="failed to get container status \"f3bc723ae3ee5c85cd2b81500071bc2c904ca453f4e15ac7412b064018301ca3\": rpc error: code = NotFound desc = could not find container \"f3bc723ae3ee5c85cd2b81500071bc2c904ca453f4e15ac7412b064018301ca3\": container with ID starting with f3bc723ae3ee5c85cd2b81500071bc2c904ca453f4e15ac7412b064018301ca3 not found: ID does not exist" Nov 03 00:04:29.759643 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:29.759605 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-915ccb-57c5454f5c-ggx2n"] Nov 03 00:04:29.807821 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:29.807788 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-915ccb-57c5454f5c-ggx2n"] Nov 03 00:04:29.884322 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:04:29.884239 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a89e95d-54df-4506-94ec-34f333911ef7" path="/var/lib/kubelet/pods/0a89e95d-54df-4506-94ec-34f333911ef7/volumes" Nov 03 00:04:57.846561 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:04:57.846521 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:05:17.391161 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:17.391053 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp"] Nov 03 00:05:17.391778 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:17.391284 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp" podUID="21dd429a-84f5-4a14-b485-51aed38a3748" containerName="maas-api" containerID="cri-o://c3cfd3c23de6be2862bf9e36cd5e971d0cab8190da13cbeb67dbcfc29512dc93" gracePeriod=30 Nov 03 00:05:17.698825 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:17.698795 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp" Nov 03 00:05:17.784762 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:17.784718 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21dd429a-84f5-4a14-b485-51aed38a3748-kube-api-access-9x8hn\" (UniqueName: \"kubernetes.io/projected/21dd429a-84f5-4a14-b485-51aed38a3748-kube-api-access-9x8hn\") pod \"21dd429a-84f5-4a14-b485-51aed38a3748\" (UID: \"21dd429a-84f5-4a14-b485-51aed38a3748\") " Nov 03 00:05:17.784967 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:17.784774 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21dd429a-84f5-4a14-b485-51aed38a3748-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21dd429a-84f5-4a14-b485-51aed38a3748-maas-api-tls\") pod \"21dd429a-84f5-4a14-b485-51aed38a3748\" (UID: \"21dd429a-84f5-4a14-b485-51aed38a3748\") " Nov 03 00:05:17.784967 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:17.784819 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21dd429a-84f5-4a14-b485-51aed38a3748-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21dd429a-84f5-4a14-b485-51aed38a3748-openshift-service-ca\") pod \"21dd429a-84f5-4a14-b485-51aed38a3748\" (UID: \"21dd429a-84f5-4a14-b485-51aed38a3748\") " Nov 03 00:05:17.785152 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:17.785113 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21dd429a-84f5-4a14-b485-51aed38a3748-metrics-certs\" (UniqueName: \"kubernetes.io/secret/21dd429a-84f5-4a14-b485-51aed38a3748-metrics-certs\") pod \"21dd429a-84f5-4a14-b485-51aed38a3748\" (UID: \"21dd429a-84f5-4a14-b485-51aed38a3748\") " Nov 03 00:05:17.785386 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:17.785342 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21dd429a-84f5-4a14-b485-51aed38a3748-openshift-service-ca" pod "21dd429a-84f5-4a14-b485-51aed38a3748" (UID: "21dd429a-84f5-4a14-b485-51aed38a3748"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:05:17.785868 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:17.785846 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21dd429a-84f5-4a14-b485-51aed38a3748-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:17.787372 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:17.787339 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21dd429a-84f5-4a14-b485-51aed38a3748-maas-api-tls" pod "21dd429a-84f5-4a14-b485-51aed38a3748" (UID: "21dd429a-84f5-4a14-b485-51aed38a3748"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:05:17.787372 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:17.787359 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21dd429a-84f5-4a14-b485-51aed38a3748-metrics-certs" pod "21dd429a-84f5-4a14-b485-51aed38a3748" (UID: "21dd429a-84f5-4a14-b485-51aed38a3748"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:05:17.787550 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:17.787528 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21dd429a-84f5-4a14-b485-51aed38a3748-kube-api-access-9x8hn" pod "21dd429a-84f5-4a14-b485-51aed38a3748" (UID: "21dd429a-84f5-4a14-b485-51aed38a3748"). InnerVolumeSpecName "kube-api-access-9x8hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:05:17.887116 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:17.887082 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x8hn\" (UniqueName: \"kubernetes.io/projected/21dd429a-84f5-4a14-b485-51aed38a3748-kube-api-access-9x8hn\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:17.887311 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:17.887119 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21dd429a-84f5-4a14-b485-51aed38a3748-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:17.887311 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:17.887178 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21dd429a-84f5-4a14-b485-51aed38a3748-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:18.014031 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:18.011231 5727 generic.go:356] "Generic (PLEG): container finished" podID="21dd429a-84f5-4a14-b485-51aed38a3748" containerID="c3cfd3c23de6be2862bf9e36cd5e971d0cab8190da13cbeb67dbcfc29512dc93" exitCode=0 Nov 03 00:05:18.014031 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:18.011330 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp" event={"ID":"21dd429a-84f5-4a14-b485-51aed38a3748","Type":"ContainerDied","Data":"c3cfd3c23de6be2862bf9e36cd5e971d0cab8190da13cbeb67dbcfc29512dc93"} Nov 03 00:05:18.014031 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:18.011439 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp" event={"ID":"21dd429a-84f5-4a14-b485-51aed38a3748","Type":"ContainerDied","Data":"9a231b4b5806ac400fb593b90ef82707ebe717f719681921aa6b167f79572eeb"} Nov 03 00:05:18.014031 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:18.011491 5727 scope.go:122] "RemoveContainer" containerID="c3cfd3c23de6be2862bf9e36cd5e971d0cab8190da13cbeb67dbcfc29512dc93" Nov 03 00:05:18.014031 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:18.011717 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp" Nov 03 00:05:18.028610 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:18.028585 5727 scope.go:122] "RemoveContainer" containerID="c3cfd3c23de6be2862bf9e36cd5e971d0cab8190da13cbeb67dbcfc29512dc93" Nov 03 00:05:18.029007 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:05:18.028978 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3cfd3c23de6be2862bf9e36cd5e971d0cab8190da13cbeb67dbcfc29512dc93\": container with ID starting with c3cfd3c23de6be2862bf9e36cd5e971d0cab8190da13cbeb67dbcfc29512dc93 not found: ID does not exist" containerID="c3cfd3c23de6be2862bf9e36cd5e971d0cab8190da13cbeb67dbcfc29512dc93" Nov 03 00:05:18.029148 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:18.029020 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3cfd3c23de6be2862bf9e36cd5e971d0cab8190da13cbeb67dbcfc29512dc93"} err="failed to get container status \"c3cfd3c23de6be2862bf9e36cd5e971d0cab8190da13cbeb67dbcfc29512dc93\": rpc error: code = NotFound desc = could not find container \"c3cfd3c23de6be2862bf9e36cd5e971d0cab8190da13cbeb67dbcfc29512dc93\": container with ID starting with c3cfd3c23de6be2862bf9e36cd5e971d0cab8190da13cbeb67dbcfc29512dc93 not found: ID does not exist" Nov 03 00:05:18.177337 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:18.177281 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp"] Nov 03 00:05:18.180590 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:18.180562 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-915ccb-555db5f75d-bgpjp"] Nov 03 00:05:19.885418 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:19.885379 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21dd429a-84f5-4a14-b485-51aed38a3748" path="/var/lib/kubelet/pods/21dd429a-84f5-4a14-b485-51aed38a3748/volumes" Nov 03 00:05:27.846707 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:05:27.846666 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:05:34.667840 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:34.667797 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r"] Nov 03 00:05:34.718532 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:34.717532 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t"] Nov 03 00:05:42.151056 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:42.151018 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ec2f9a-589bcfb455-2r9sr"] Nov 03 00:05:42.253318 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:42.253284 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ec2f9a-76969d66d7-6b8dx"] Nov 03 00:05:47.229961 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:47.229892 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-ec2f9a-589bcfb455-2r9sr" podUID="c483a352-ae64-4bf1-bc45-6950f022114d" containerName="payload-processing" containerID="cri-o://c1787a0827f8e581b953ddd49a631dee38bb09ecd2ff03529dee301cbb3a500c" gracePeriod=25 Nov 03 00:05:47.253783 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:47.253738 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ec2f9a-76969d66d7-6b8dx" podUID="1cdb3d15-9771-4a60-8e05-66895477704a" containerName="payload-pre-processing" containerID="cri-o://66fe2597719c466391725b0c5c7442dca469ecae625c17e5969c73a8c558ba3b" gracePeriod=25 Nov 03 00:05:49.699225 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:49.699178 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" podUID="cdc40e13-d236-4739-a4f5-098aea3fb308" containerName="main" containerID="cri-o://caae0e836faa6edc9ef88cb87cbeddf92a2f10aec95d2fa6ed3ae9b7b65c4d28" gracePeriod=45 Nov 03 00:05:49.741212 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:49.741171 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" podUID="9a46665a-4362-4a43-8320-7dedd086b8d5" containerName="main" containerID="cri-o://3bd2a74951b111dbb2369d5192c9c521c7268cd78f91af6ca43cb803cd68ec46" gracePeriod=45 Nov 03 00:05:50.027383 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.027356 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" Nov 03 00:05:50.074419 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.074392 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" Nov 03 00:05:50.125928 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.125895 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a46665a-4362-4a43-8320-7dedd086b8d5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a46665a-4362-4a43-8320-7dedd086b8d5-dshm\") pod \"9a46665a-4362-4a43-8320-7dedd086b8d5\" (UID: \"9a46665a-4362-4a43-8320-7dedd086b8d5\") " Nov 03 00:05:50.126106 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.125938 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc40e13-d236-4739-a4f5-098aea3fb308-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc40e13-d236-4739-a4f5-098aea3fb308-tmp-dir\") pod \"cdc40e13-d236-4739-a4f5-098aea3fb308\" (UID: \"cdc40e13-d236-4739-a4f5-098aea3fb308\") " Nov 03 00:05:50.126106 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.125964 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a46665a-4362-4a43-8320-7dedd086b8d5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a46665a-4362-4a43-8320-7dedd086b8d5-model-cache\") pod \"9a46665a-4362-4a43-8320-7dedd086b8d5\" (UID: \"9a46665a-4362-4a43-8320-7dedd086b8d5\") " Nov 03 00:05:50.126106 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.126005 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdc40e13-d236-4739-a4f5-098aea3fb308-kube-api-access-7nnb4\" (UniqueName: \"kubernetes.io/projected/cdc40e13-d236-4739-a4f5-098aea3fb308-kube-api-access-7nnb4\") pod \"cdc40e13-d236-4739-a4f5-098aea3fb308\" (UID: \"cdc40e13-d236-4739-a4f5-098aea3fb308\") " Nov 03 00:05:50.126106 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.126077 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdc40e13-d236-4739-a4f5-098aea3fb308-tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc40e13-d236-4739-a4f5-098aea3fb308-tls-certs\") pod \"cdc40e13-d236-4739-a4f5-098aea3fb308\" (UID: \"cdc40e13-d236-4739-a4f5-098aea3fb308\") " Nov 03 00:05:50.126386 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.126122 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a46665a-4362-4a43-8320-7dedd086b8d5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a46665a-4362-4a43-8320-7dedd086b8d5-tmp-dir\") pod \"9a46665a-4362-4a43-8320-7dedd086b8d5\" (UID: \"9a46665a-4362-4a43-8320-7dedd086b8d5\") " Nov 03 00:05:50.126386 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.126213 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc40e13-d236-4739-a4f5-098aea3fb308-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc40e13-d236-4739-a4f5-098aea3fb308-model-cache\") pod \"cdc40e13-d236-4739-a4f5-098aea3fb308\" (UID: \"cdc40e13-d236-4739-a4f5-098aea3fb308\") " Nov 03 00:05:50.126386 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.126235 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a46665a-4362-4a43-8320-7dedd086b8d5-kube-api-access-9m7c9\" (UniqueName: \"kubernetes.io/projected/9a46665a-4362-4a43-8320-7dedd086b8d5-kube-api-access-9m7c9\") pod \"9a46665a-4362-4a43-8320-7dedd086b8d5\" (UID: \"9a46665a-4362-4a43-8320-7dedd086b8d5\") " Nov 03 00:05:50.126386 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.126262 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a46665a-4362-4a43-8320-7dedd086b8d5-home\" (UniqueName: \"kubernetes.io/empty-dir/9a46665a-4362-4a43-8320-7dedd086b8d5-home\") pod \"9a46665a-4362-4a43-8320-7dedd086b8d5\" (UID: \"9a46665a-4362-4a43-8320-7dedd086b8d5\") " Nov 03 00:05:50.126386 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.126272 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc40e13-d236-4739-a4f5-098aea3fb308-tmp-dir" pod "cdc40e13-d236-4739-a4f5-098aea3fb308" (UID: "cdc40e13-d236-4739-a4f5-098aea3fb308"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:50.126386 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.126360 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc40e13-d236-4739-a4f5-098aea3fb308-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc40e13-d236-4739-a4f5-098aea3fb308-dshm\") pod \"cdc40e13-d236-4739-a4f5-098aea3fb308\" (UID: \"cdc40e13-d236-4739-a4f5-098aea3fb308\") " Nov 03 00:05:50.126744 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.126400 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a46665a-4362-4a43-8320-7dedd086b8d5-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a46665a-4362-4a43-8320-7dedd086b8d5-tls-certs\") pod \"9a46665a-4362-4a43-8320-7dedd086b8d5\" (UID: \"9a46665a-4362-4a43-8320-7dedd086b8d5\") " Nov 03 00:05:50.126744 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.126465 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc40e13-d236-4739-a4f5-098aea3fb308-home\" (UniqueName: \"kubernetes.io/empty-dir/cdc40e13-d236-4739-a4f5-098aea3fb308-home\") pod \"cdc40e13-d236-4739-a4f5-098aea3fb308\" (UID: \"cdc40e13-d236-4739-a4f5-098aea3fb308\") " Nov 03 00:05:50.126744 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.126497 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a46665a-4362-4a43-8320-7dedd086b8d5-tmp-dir" pod "9a46665a-4362-4a43-8320-7dedd086b8d5" (UID: "9a46665a-4362-4a43-8320-7dedd086b8d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:50.126744 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.126516 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a46665a-4362-4a43-8320-7dedd086b8d5-home" pod "9a46665a-4362-4a43-8320-7dedd086b8d5" (UID: "9a46665a-4362-4a43-8320-7dedd086b8d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:50.126744 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.126274 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a46665a-4362-4a43-8320-7dedd086b8d5-model-cache" pod "9a46665a-4362-4a43-8320-7dedd086b8d5" (UID: "9a46665a-4362-4a43-8320-7dedd086b8d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:50.126744 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.126552 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc40e13-d236-4739-a4f5-098aea3fb308-model-cache" pod "cdc40e13-d236-4739-a4f5-098aea3fb308" (UID: "cdc40e13-d236-4739-a4f5-098aea3fb308"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:50.127372 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.126818 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc40e13-d236-4739-a4f5-098aea3fb308-home" pod "cdc40e13-d236-4739-a4f5-098aea3fb308" (UID: "cdc40e13-d236-4739-a4f5-098aea3fb308"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:50.127372 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.127024 5727 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a46665a-4362-4a43-8320-7dedd086b8d5-tmp-dir\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:50.127372 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.127045 5727 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc40e13-d236-4739-a4f5-098aea3fb308-model-cache\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:50.127372 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.127060 5727 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a46665a-4362-4a43-8320-7dedd086b8d5-home\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:50.127372 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.127094 5727 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc40e13-d236-4739-a4f5-098aea3fb308-home\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:50.127372 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.127212 5727 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc40e13-d236-4739-a4f5-098aea3fb308-tmp-dir\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:50.127372 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.127278 5727 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a46665a-4362-4a43-8320-7dedd086b8d5-model-cache\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:50.128773 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.128746 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc40e13-d236-4739-a4f5-098aea3fb308-kube-api-access-7nnb4" pod "cdc40e13-d236-4739-a4f5-098aea3fb308" (UID: "cdc40e13-d236-4739-a4f5-098aea3fb308"). InnerVolumeSpecName "kube-api-access-7nnb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:05:50.128886 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.128785 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc40e13-d236-4739-a4f5-098aea3fb308-dshm" pod "cdc40e13-d236-4739-a4f5-098aea3fb308" (UID: "cdc40e13-d236-4739-a4f5-098aea3fb308"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:50.128886 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.128871 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a46665a-4362-4a43-8320-7dedd086b8d5-dshm" pod "9a46665a-4362-4a43-8320-7dedd086b8d5" (UID: "9a46665a-4362-4a43-8320-7dedd086b8d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:50.129293 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.129271 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a46665a-4362-4a43-8320-7dedd086b8d5-kube-api-access-9m7c9" pod "9a46665a-4362-4a43-8320-7dedd086b8d5" (UID: "9a46665a-4362-4a43-8320-7dedd086b8d5"). InnerVolumeSpecName "kube-api-access-9m7c9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:05:50.129383 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.129306 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdc40e13-d236-4739-a4f5-098aea3fb308-tls-certs" pod "cdc40e13-d236-4739-a4f5-098aea3fb308" (UID: "cdc40e13-d236-4739-a4f5-098aea3fb308"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:05:50.129552 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.129465 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a46665a-4362-4a43-8320-7dedd086b8d5-tls-certs" pod "9a46665a-4362-4a43-8320-7dedd086b8d5" (UID: "9a46665a-4362-4a43-8320-7dedd086b8d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:05:50.228679 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.228636 5727 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a46665a-4362-4a43-8320-7dedd086b8d5-dshm\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:50.228875 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.228685 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nnb4\" (UniqueName: \"kubernetes.io/projected/cdc40e13-d236-4739-a4f5-098aea3fb308-kube-api-access-7nnb4\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:50.228875 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.228703 5727 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc40e13-d236-4739-a4f5-098aea3fb308-tls-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:50.228875 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.228719 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m7c9\" (UniqueName: \"kubernetes.io/projected/9a46665a-4362-4a43-8320-7dedd086b8d5-kube-api-access-9m7c9\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:50.228875 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.228733 5727 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc40e13-d236-4739-a4f5-098aea3fb308-dshm\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:50.228875 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.228747 5727 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a46665a-4362-4a43-8320-7dedd086b8d5-tls-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:50.300253 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.295880 5727 generic.go:356] "Generic (PLEG): container finished" podID="cdc40e13-d236-4739-a4f5-098aea3fb308" containerID="caae0e836faa6edc9ef88cb87cbeddf92a2f10aec95d2fa6ed3ae9b7b65c4d28" exitCode=0 Nov 03 00:05:50.300253 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.295954 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" Nov 03 00:05:50.300253 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.295970 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" event={"ID":"cdc40e13-d236-4739-a4f5-098aea3fb308","Type":"ContainerDied","Data":"caae0e836faa6edc9ef88cb87cbeddf92a2f10aec95d2fa6ed3ae9b7b65c4d28"} Nov 03 00:05:50.300253 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.296043 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r" event={"ID":"cdc40e13-d236-4739-a4f5-098aea3fb308","Type":"ContainerDied","Data":"847d0e11b487c5e02f28c73ec1b425137257f4c6962fa8bc8c69906b6e471740"} Nov 03 00:05:50.300253 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.296069 5727 scope.go:122] "RemoveContainer" containerID="caae0e836faa6edc9ef88cb87cbeddf92a2f10aec95d2fa6ed3ae9b7b65c4d28" Nov 03 00:05:50.300253 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.298497 5727 generic.go:356] "Generic (PLEG): container finished" podID="9a46665a-4362-4a43-8320-7dedd086b8d5" containerID="3bd2a74951b111dbb2369d5192c9c521c7268cd78f91af6ca43cb803cd68ec46" exitCode=0 Nov 03 00:05:50.300253 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.298700 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" Nov 03 00:05:50.300253 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.299029 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" event={"ID":"9a46665a-4362-4a43-8320-7dedd086b8d5","Type":"ContainerDied","Data":"3bd2a74951b111dbb2369d5192c9c521c7268cd78f91af6ca43cb803cd68ec46"} Nov 03 00:05:50.300253 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.299070 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" event={"ID":"9a46665a-4362-4a43-8320-7dedd086b8d5","Type":"ContainerDied","Data":"736e2b6736892517bf5d523093a3cebb02a621ea48f8bfd36325c2a39c51f151"} Nov 03 00:05:50.312608 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.312580 5727 scope.go:122] "RemoveContainer" containerID="caae0e836faa6edc9ef88cb87cbeddf92a2f10aec95d2fa6ed3ae9b7b65c4d28" Nov 03 00:05:50.312976 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:05:50.312954 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caae0e836faa6edc9ef88cb87cbeddf92a2f10aec95d2fa6ed3ae9b7b65c4d28\": container with ID starting with caae0e836faa6edc9ef88cb87cbeddf92a2f10aec95d2fa6ed3ae9b7b65c4d28 not found: ID does not exist" containerID="caae0e836faa6edc9ef88cb87cbeddf92a2f10aec95d2fa6ed3ae9b7b65c4d28" Nov 03 00:05:50.313093 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.312988 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caae0e836faa6edc9ef88cb87cbeddf92a2f10aec95d2fa6ed3ae9b7b65c4d28"} err="failed to get container status \"caae0e836faa6edc9ef88cb87cbeddf92a2f10aec95d2fa6ed3ae9b7b65c4d28\": rpc error: code = NotFound desc = could not find container \"caae0e836faa6edc9ef88cb87cbeddf92a2f10aec95d2fa6ed3ae9b7b65c4d28\": container with ID starting with caae0e836faa6edc9ef88cb87cbeddf92a2f10aec95d2fa6ed3ae9b7b65c4d28 not found: ID does not exist" Nov 03 00:05:50.313093 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.313014 5727 scope.go:122] "RemoveContainer" containerID="3bd2a74951b111dbb2369d5192c9c521c7268cd78f91af6ca43cb803cd68ec46" Nov 03 00:05:50.327214 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.327187 5727 scope.go:122] "RemoveContainer" containerID="3bd2a74951b111dbb2369d5192c9c521c7268cd78f91af6ca43cb803cd68ec46" Nov 03 00:05:50.327647 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:05:50.327618 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bd2a74951b111dbb2369d5192c9c521c7268cd78f91af6ca43cb803cd68ec46\": container with ID starting with 3bd2a74951b111dbb2369d5192c9c521c7268cd78f91af6ca43cb803cd68ec46 not found: ID does not exist" containerID="3bd2a74951b111dbb2369d5192c9c521c7268cd78f91af6ca43cb803cd68ec46" Nov 03 00:05:50.327741 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.327662 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bd2a74951b111dbb2369d5192c9c521c7268cd78f91af6ca43cb803cd68ec46"} err="failed to get container status \"3bd2a74951b111dbb2369d5192c9c521c7268cd78f91af6ca43cb803cd68ec46\": rpc error: code = NotFound desc = could not find container \"3bd2a74951b111dbb2369d5192c9c521c7268cd78f91af6ca43cb803cd68ec46\": container with ID starting with 3bd2a74951b111dbb2369d5192c9c521c7268cd78f91af6ca43cb803cd68ec46 not found: ID does not exist" Nov 03 00:05:50.462899 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.462862 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r"] Nov 03 00:05:50.466991 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.466954 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ec2f9a/distinct-2-w0-ec2f9a-kserve-9768449b4-p5h7r"] Nov 03 00:05:50.576848 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.576808 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t"] Nov 03 00:05:50.581964 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:50.581935 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t"] Nov 03 00:05:51.031212 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:51.031085 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ec2f9a/distinct-w0-ec2f9a-kserve-5fcc8b5995-hhg7t" podUID="9a46665a-4362-4a43-8320-7dedd086b8d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Nov 03 00:05:51.886004 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:51.885967 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a46665a-4362-4a43-8320-7dedd086b8d5" path="/var/lib/kubelet/pods/9a46665a-4362-4a43-8320-7dedd086b8d5/volumes" Nov 03 00:05:51.886433 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:51.886417 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdc40e13-d236-4739-a4f5-098aea3fb308" path="/var/lib/kubelet/pods/cdc40e13-d236-4739-a4f5-098aea3fb308/volumes" Nov 03 00:05:55.711235 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:55.711192 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-ec2f9a-589bcfb455-2r9sr" podUID="c483a352-ae64-4bf1-bc45-6950f022114d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:05:55.711235 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:05:55.711237 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ec2f9a-76969d66d7-6b8dx" podUID="1cdb3d15-9771-4a60-8e05-66895477704a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:05:57.848070 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:05:57.848028 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:06:05.711113 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:05.711077 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ec2f9a-76969d66d7-6b8dx" podUID="1cdb3d15-9771-4a60-8e05-66895477704a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:06:05.711565 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:05.711157 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-ec2f9a-589bcfb455-2r9sr" podUID="c483a352-ae64-4bf1-bc45-6950f022114d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:06:07.445456 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:07.443844 5727 generic.go:356] "Generic (PLEG): container finished" podID="1cdb3d15-9771-4a60-8e05-66895477704a" containerID="66fe2597719c466391725b0c5c7442dca469ecae625c17e5969c73a8c558ba3b" exitCode=0 Nov 03 00:06:07.454340 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:07.453244 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ec2f9a-76969d66d7-6b8dx" event={"ID":"1cdb3d15-9771-4a60-8e05-66895477704a","Type":"ContainerDied","Data":"66fe2597719c466391725b0c5c7442dca469ecae625c17e5969c73a8c558ba3b"} Nov 03 00:06:07.460803 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:07.460768 5727 generic.go:356] "Generic (PLEG): container finished" podID="c483a352-ae64-4bf1-bc45-6950f022114d" containerID="c1787a0827f8e581b953ddd49a631dee38bb09ecd2ff03529dee301cbb3a500c" exitCode=0 Nov 03 00:06:07.460948 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:07.460814 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ec2f9a-589bcfb455-2r9sr" event={"ID":"c483a352-ae64-4bf1-bc45-6950f022114d","Type":"ContainerDied","Data":"c1787a0827f8e581b953ddd49a631dee38bb09ecd2ff03529dee301cbb3a500c"} Nov 03 00:06:07.544482 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:07.544452 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ec2f9a-589bcfb455-2r9sr" Nov 03 00:06:07.587248 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:07.587224 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ec2f9a-76969d66d7-6b8dx" Nov 03 00:06:07.642034 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:07.642007 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c483a352-ae64-4bf1-bc45-6950f022114d-kube-api-access-rs6qg\" (UniqueName: \"kubernetes.io/projected/c483a352-ae64-4bf1-bc45-6950f022114d-kube-api-access-rs6qg\") pod \"c483a352-ae64-4bf1-bc45-6950f022114d\" (UID: \"c483a352-ae64-4bf1-bc45-6950f022114d\") " Nov 03 00:06:07.642265 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:07.642244 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c483a352-ae64-4bf1-bc45-6950f022114d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c483a352-ae64-4bf1-bc45-6950f022114d-plugins-config-volume\") pod \"c483a352-ae64-4bf1-bc45-6950f022114d\" (UID: \"c483a352-ae64-4bf1-bc45-6950f022114d\") " Nov 03 00:06:07.642618 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:07.642591 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c483a352-ae64-4bf1-bc45-6950f022114d-plugins-config-volume" pod "c483a352-ae64-4bf1-bc45-6950f022114d" (UID: "c483a352-ae64-4bf1-bc45-6950f022114d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:07.642757 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:07.642741 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c483a352-ae64-4bf1-bc45-6950f022114d-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:07.644456 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:07.644438 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c483a352-ae64-4bf1-bc45-6950f022114d-kube-api-access-rs6qg" pod "c483a352-ae64-4bf1-bc45-6950f022114d" (UID: "c483a352-ae64-4bf1-bc45-6950f022114d"). InnerVolumeSpecName "kube-api-access-rs6qg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:07.743961 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:07.743929 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cdb3d15-9771-4a60-8e05-66895477704a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cdb3d15-9771-4a60-8e05-66895477704a-plugins-config-volume\") pod \"1cdb3d15-9771-4a60-8e05-66895477704a\" (UID: \"1cdb3d15-9771-4a60-8e05-66895477704a\") " Nov 03 00:06:07.744181 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:07.744044 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cdb3d15-9771-4a60-8e05-66895477704a-kube-api-access-sjjzh\" (UniqueName: \"kubernetes.io/projected/1cdb3d15-9771-4a60-8e05-66895477704a-kube-api-access-sjjzh\") pod \"1cdb3d15-9771-4a60-8e05-66895477704a\" (UID: \"1cdb3d15-9771-4a60-8e05-66895477704a\") " Nov 03 00:06:07.744357 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:07.744332 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cdb3d15-9771-4a60-8e05-66895477704a-plugins-config-volume" pod "1cdb3d15-9771-4a60-8e05-66895477704a" (UID: "1cdb3d15-9771-4a60-8e05-66895477704a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:07.744620 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:07.744602 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cdb3d15-9771-4a60-8e05-66895477704a-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:07.744664 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:07.744624 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs6qg\" (UniqueName: \"kubernetes.io/projected/c483a352-ae64-4bf1-bc45-6950f022114d-kube-api-access-rs6qg\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:07.746401 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:07.746377 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cdb3d15-9771-4a60-8e05-66895477704a-kube-api-access-sjjzh" pod "1cdb3d15-9771-4a60-8e05-66895477704a" (UID: "1cdb3d15-9771-4a60-8e05-66895477704a"). InnerVolumeSpecName "kube-api-access-sjjzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:07.846310 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:07.846280 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjjzh\" (UniqueName: \"kubernetes.io/projected/1cdb3d15-9771-4a60-8e05-66895477704a-kube-api-access-sjjzh\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:08.470672 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:08.470637 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ec2f9a-589bcfb455-2r9sr" Nov 03 00:06:08.471198 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:08.470674 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ec2f9a-589bcfb455-2r9sr" event={"ID":"c483a352-ae64-4bf1-bc45-6950f022114d","Type":"ContainerDied","Data":"0e3672d8b8e278d5eafb72e7e5b14b827331265843fd30d66ac29ee21c9a2af4"} Nov 03 00:06:08.471198 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:08.470745 5727 scope.go:122] "RemoveContainer" containerID="c1787a0827f8e581b953ddd49a631dee38bb09ecd2ff03529dee301cbb3a500c" Nov 03 00:06:08.472487 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:08.472470 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ec2f9a-76969d66d7-6b8dx" Nov 03 00:06:08.472599 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:08.472488 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ec2f9a-76969d66d7-6b8dx" event={"ID":"1cdb3d15-9771-4a60-8e05-66895477704a","Type":"ContainerDied","Data":"40dd4136b38074f14e472d81603a1c9ca73adce298851e8229769310477107b9"} Nov 03 00:06:08.484073 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:08.484056 5727 scope.go:122] "RemoveContainer" containerID="66fe2597719c466391725b0c5c7442dca469ecae625c17e5969c73a8c558ba3b" Nov 03 00:06:08.635109 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:08.635082 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ec2f9a-589bcfb455-2r9sr"] Nov 03 00:06:08.685054 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:08.685021 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ec2f9a-589bcfb455-2r9sr"] Nov 03 00:06:08.749171 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:08.749141 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ec2f9a-76969d66d7-6b8dx"] Nov 03 00:06:08.756946 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:08.756915 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ec2f9a-76969d66d7-6b8dx"] Nov 03 00:06:09.886363 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:09.886328 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cdb3d15-9771-4a60-8e05-66895477704a" path="/var/lib/kubelet/pods/1cdb3d15-9771-4a60-8e05-66895477704a/volumes" Nov 03 00:06:09.886900 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:09.886881 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c483a352-ae64-4bf1-bc45-6950f022114d" path="/var/lib/kubelet/pods/c483a352-ae64-4bf1-bc45-6950f022114d/volumes" Nov 03 00:06:27.846295 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:06:27.846242 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:06:40.574221 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:40.574186 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc"] Nov 03 00:06:40.574778 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:40.574460 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc" podUID="309a0541-c36f-4914-b5b4-80b263b66f5e" containerName="maas-api" containerID="cri-o://c55917c0200acd443038cad7afdd58affca936b53bf78879137c6f221694d64b" gracePeriod=30 Nov 03 00:06:40.615887 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:06:40.615845 5727 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod309a0541_c36f_4914_b5b4_80b263b66f5e.slice/crio-conmon-c55917c0200acd443038cad7afdd58affca936b53bf78879137c6f221694d64b.scope\": RecentStats: unable to find data in memory cache]" Nov 03 00:06:40.720827 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:40.720787 5727 generic.go:356] "Generic (PLEG): container finished" podID="309a0541-c36f-4914-b5b4-80b263b66f5e" containerID="c55917c0200acd443038cad7afdd58affca936b53bf78879137c6f221694d64b" exitCode=0 Nov 03 00:06:40.721031 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:40.720865 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc" event={"ID":"309a0541-c36f-4914-b5b4-80b263b66f5e","Type":"ContainerDied","Data":"c55917c0200acd443038cad7afdd58affca936b53bf78879137c6f221694d64b"} Nov 03 00:06:40.867039 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:40.867013 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc" Nov 03 00:06:40.951266 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:40.951237 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/309a0541-c36f-4914-b5b4-80b263b66f5e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/309a0541-c36f-4914-b5b4-80b263b66f5e-metrics-certs\") pod \"309a0541-c36f-4914-b5b4-80b263b66f5e\" (UID: \"309a0541-c36f-4914-b5b4-80b263b66f5e\") " Nov 03 00:06:40.951425 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:40.951275 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/309a0541-c36f-4914-b5b4-80b263b66f5e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/309a0541-c36f-4914-b5b4-80b263b66f5e-openshift-service-ca\") pod \"309a0541-c36f-4914-b5b4-80b263b66f5e\" (UID: \"309a0541-c36f-4914-b5b4-80b263b66f5e\") " Nov 03 00:06:40.951425 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:40.951382 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/309a0541-c36f-4914-b5b4-80b263b66f5e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/309a0541-c36f-4914-b5b4-80b263b66f5e-maas-api-tls\") pod \"309a0541-c36f-4914-b5b4-80b263b66f5e\" (UID: \"309a0541-c36f-4914-b5b4-80b263b66f5e\") " Nov 03 00:06:40.951558 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:40.951449 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/309a0541-c36f-4914-b5b4-80b263b66f5e-kube-api-access-7p4w9\" (UniqueName: \"kubernetes.io/projected/309a0541-c36f-4914-b5b4-80b263b66f5e-kube-api-access-7p4w9\") pod \"309a0541-c36f-4914-b5b4-80b263b66f5e\" (UID: \"309a0541-c36f-4914-b5b4-80b263b66f5e\") " Nov 03 00:06:40.952362 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:40.952053 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/309a0541-c36f-4914-b5b4-80b263b66f5e-openshift-service-ca" pod "309a0541-c36f-4914-b5b4-80b263b66f5e" (UID: "309a0541-c36f-4914-b5b4-80b263b66f5e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:40.952965 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:40.952939 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/309a0541-c36f-4914-b5b4-80b263b66f5e-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:40.956811 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:40.954946 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/309a0541-c36f-4914-b5b4-80b263b66f5e-metrics-certs" pod "309a0541-c36f-4914-b5b4-80b263b66f5e" (UID: "309a0541-c36f-4914-b5b4-80b263b66f5e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:40.956811 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:40.955303 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/309a0541-c36f-4914-b5b4-80b263b66f5e-maas-api-tls" pod "309a0541-c36f-4914-b5b4-80b263b66f5e" (UID: "309a0541-c36f-4914-b5b4-80b263b66f5e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:40.972748 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:40.972693 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/309a0541-c36f-4914-b5b4-80b263b66f5e-kube-api-access-7p4w9" pod "309a0541-c36f-4914-b5b4-80b263b66f5e" (UID: "309a0541-c36f-4914-b5b4-80b263b66f5e"). InnerVolumeSpecName "kube-api-access-7p4w9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:41.054525 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:41.054481 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309a0541-c36f-4914-b5b4-80b263b66f5e-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:41.054525 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:41.054522 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/309a0541-c36f-4914-b5b4-80b263b66f5e-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:41.054732 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:41.054539 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p4w9\" (UniqueName: \"kubernetes.io/projected/309a0541-c36f-4914-b5b4-80b263b66f5e-kube-api-access-7p4w9\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:41.729144 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:41.729099 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc" Nov 03 00:06:41.729643 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:41.729156 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc" event={"ID":"309a0541-c36f-4914-b5b4-80b263b66f5e","Type":"ContainerDied","Data":"ded9c02bb8b404e46e67b2dd72f3564866f085d9326cb9e2a27616761ce2b725"} Nov 03 00:06:41.729643 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:41.729251 5727 scope.go:122] "RemoveContainer" containerID="c55917c0200acd443038cad7afdd58affca936b53bf78879137c6f221694d64b" Nov 03 00:06:41.893834 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:41.893801 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc"] Nov 03 00:06:41.899358 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:41.899334 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ec2f9a-56cccc6b8c-gfzdc"] Nov 03 00:06:43.886176 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:43.886064 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="309a0541-c36f-4914-b5b4-80b263b66f5e" path="/var/lib/kubelet/pods/309a0541-c36f-4914-b5b4-80b263b66f5e/volumes" Nov 03 00:06:52.613445 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.613400 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7"] Nov 03 00:06:52.614312 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.614285 5727 cpu_manager.go:414] "containerMap: removing container" podUID="cdc40e13-d236-4739-a4f5-098aea3fb308" containerName="main" Nov 03 00:06:52.614485 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.614316 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdc40e13-d236-4739-a4f5-098aea3fb308" containerName="main" Nov 03 00:06:52.614485 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.614338 5727 cpu_manager.go:414] "containerMap: removing container" podUID="0a89e95d-54df-4506-94ec-34f333911ef7" containerName="payload-processing" Nov 03 00:06:52.614485 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.614350 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a89e95d-54df-4506-94ec-34f333911ef7" containerName="payload-processing" Nov 03 00:06:52.614485 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.614365 5727 cpu_manager.go:414] "containerMap: removing container" podUID="21dd429a-84f5-4a14-b485-51aed38a3748" containerName="maas-api" Nov 03 00:06:52.614485 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.614377 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21dd429a-84f5-4a14-b485-51aed38a3748" containerName="maas-api" Nov 03 00:06:52.614485 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.614391 5727 cpu_manager.go:414] "containerMap: removing container" podUID="9a46665a-4362-4a43-8320-7dedd086b8d5" containerName="main" Nov 03 00:06:52.614485 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.614399 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a46665a-4362-4a43-8320-7dedd086b8d5" containerName="main" Nov 03 00:06:52.614485 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.614419 5727 cpu_manager.go:414] "containerMap: removing container" podUID="c483a352-ae64-4bf1-bc45-6950f022114d" containerName="payload-processing" Nov 03 00:06:52.614485 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.614427 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c483a352-ae64-4bf1-bc45-6950f022114d" containerName="payload-processing" Nov 03 00:06:52.614485 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.614450 5727 cpu_manager.go:414] "containerMap: removing container" podUID="1d559e50-9c39-42dd-8ac1-6d42ebb726c4" containerName="payload-processing" Nov 03 00:06:52.614485 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.614457 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d559e50-9c39-42dd-8ac1-6d42ebb726c4" containerName="payload-processing" Nov 03 00:06:52.614485 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.614486 5727 cpu_manager.go:414] "containerMap: removing container" podUID="4b213bcf-90cf-414e-9432-e7887fa901fa" containerName="payload-processing" Nov 03 00:06:52.614485 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.614493 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b213bcf-90cf-414e-9432-e7887fa901fa" containerName="payload-processing" Nov 03 00:06:52.615294 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.614510 5727 cpu_manager.go:414] "containerMap: removing container" podUID="309a0541-c36f-4914-b5b4-80b263b66f5e" containerName="maas-api" Nov 03 00:06:52.615294 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.614517 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="309a0541-c36f-4914-b5b4-80b263b66f5e" containerName="maas-api" Nov 03 00:06:52.615294 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.614538 5727 cpu_manager.go:414] "containerMap: removing container" podUID="1cdb3d15-9771-4a60-8e05-66895477704a" containerName="payload-pre-processing" Nov 03 00:06:52.615294 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.614546 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cdb3d15-9771-4a60-8e05-66895477704a" containerName="payload-pre-processing" Nov 03 00:06:52.615294 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.614712 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="c483a352-ae64-4bf1-bc45-6950f022114d" containerName="payload-processing" Nov 03 00:06:52.615294 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.614726 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d559e50-9c39-42dd-8ac1-6d42ebb726c4" containerName="payload-processing" Nov 03 00:06:52.615294 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.614739 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b213bcf-90cf-414e-9432-e7887fa901fa" containerName="payload-processing" Nov 03 00:06:52.615294 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.614749 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="309a0541-c36f-4914-b5b4-80b263b66f5e" containerName="maas-api" Nov 03 00:06:52.615294 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.614785 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cdb3d15-9771-4a60-8e05-66895477704a" containerName="payload-pre-processing" Nov 03 00:06:52.615294 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.614800 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdc40e13-d236-4739-a4f5-098aea3fb308" containerName="main" Nov 03 00:06:52.615294 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.614810 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a89e95d-54df-4506-94ec-34f333911ef7" containerName="payload-processing" Nov 03 00:06:52.615294 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.614818 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="21dd429a-84f5-4a14-b485-51aed38a3748" containerName="maas-api" Nov 03 00:06:52.615294 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.614830 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a46665a-4362-4a43-8320-7dedd086b8d5" containerName="main" Nov 03 00:06:52.618903 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.618880 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" Nov 03 00:06:52.675219 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.675188 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7"] Nov 03 00:06:52.697770 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.697737 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f739ec5f-a443-44c4-94d6-4077616af9c7-home\") pod \"rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7\" (UID: \"f739ec5f-a443-44c4-94d6-4077616af9c7\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" Nov 03 00:06:52.697952 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.697924 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbq5l\" (UniqueName: \"kubernetes.io/projected/f739ec5f-a443-44c4-94d6-4077616af9c7-kube-api-access-jbq5l\") pod \"rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7\" (UID: \"f739ec5f-a443-44c4-94d6-4077616af9c7\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" Nov 03 00:06:52.698189 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.698162 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f739ec5f-a443-44c4-94d6-4077616af9c7-model-cache\") pod \"rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7\" (UID: \"f739ec5f-a443-44c4-94d6-4077616af9c7\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" Nov 03 00:06:52.698259 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.698207 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f739ec5f-a443-44c4-94d6-4077616af9c7-dshm\") pod \"rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7\" (UID: \"f739ec5f-a443-44c4-94d6-4077616af9c7\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" Nov 03 00:06:52.698259 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.698234 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f739ec5f-a443-44c4-94d6-4077616af9c7-tmp-dir\") pod \"rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7\" (UID: \"f739ec5f-a443-44c4-94d6-4077616af9c7\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" Nov 03 00:06:52.698340 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.698262 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f739ec5f-a443-44c4-94d6-4077616af9c7-tls-certs\") pod \"rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7\" (UID: \"f739ec5f-a443-44c4-94d6-4077616af9c7\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" Nov 03 00:06:52.720294 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.720268 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-3bee1b\"/\"default-dockercfg-j75g4\"" Nov 03 00:06:52.720416 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.720317 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-3bee1b\"/\"rate-test-model-3bee1b-kserve-self-signed-certs\"" Nov 03 00:06:52.720523 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.720503 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-3bee1b\"/\"kube-root-ca.crt\"" Nov 03 00:06:52.720523 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.720518 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-3bee1b\"/\"openshift-service-ca.crt\"" Nov 03 00:06:52.799309 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.799276 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbq5l\" (UniqueName: \"kubernetes.io/projected/f739ec5f-a443-44c4-94d6-4077616af9c7-kube-api-access-jbq5l\") pod \"rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7\" (UID: \"f739ec5f-a443-44c4-94d6-4077616af9c7\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" Nov 03 00:06:52.799531 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.799402 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f739ec5f-a443-44c4-94d6-4077616af9c7-model-cache\") pod \"rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7\" (UID: \"f739ec5f-a443-44c4-94d6-4077616af9c7\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" Nov 03 00:06:52.799531 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.799424 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f739ec5f-a443-44c4-94d6-4077616af9c7-dshm\") pod \"rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7\" (UID: \"f739ec5f-a443-44c4-94d6-4077616af9c7\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" Nov 03 00:06:52.799531 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.799440 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f739ec5f-a443-44c4-94d6-4077616af9c7-tmp-dir\") pod \"rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7\" (UID: \"f739ec5f-a443-44c4-94d6-4077616af9c7\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" Nov 03 00:06:52.799531 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.799456 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f739ec5f-a443-44c4-94d6-4077616af9c7-tls-certs\") pod \"rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7\" (UID: \"f739ec5f-a443-44c4-94d6-4077616af9c7\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" Nov 03 00:06:52.799801 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.799656 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f739ec5f-a443-44c4-94d6-4077616af9c7-home\") pod \"rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7\" (UID: \"f739ec5f-a443-44c4-94d6-4077616af9c7\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" Nov 03 00:06:52.800061 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.800037 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f739ec5f-a443-44c4-94d6-4077616af9c7-tmp-dir\") pod \"rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7\" (UID: \"f739ec5f-a443-44c4-94d6-4077616af9c7\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" Nov 03 00:06:52.800201 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.800077 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f739ec5f-a443-44c4-94d6-4077616af9c7-home\") pod \"rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7\" (UID: \"f739ec5f-a443-44c4-94d6-4077616af9c7\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" Nov 03 00:06:52.800201 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.800177 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f739ec5f-a443-44c4-94d6-4077616af9c7-model-cache\") pod \"rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7\" (UID: \"f739ec5f-a443-44c4-94d6-4077616af9c7\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" Nov 03 00:06:52.801991 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.801968 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f739ec5f-a443-44c4-94d6-4077616af9c7-dshm\") pod \"rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7\" (UID: \"f739ec5f-a443-44c4-94d6-4077616af9c7\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" Nov 03 00:06:52.802285 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.802269 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f739ec5f-a443-44c4-94d6-4077616af9c7-tls-certs\") pod \"rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7\" (UID: \"f739ec5f-a443-44c4-94d6-4077616af9c7\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" Nov 03 00:06:52.860679 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.860644 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbq5l\" (UniqueName: \"kubernetes.io/projected/f739ec5f-a443-44c4-94d6-4077616af9c7-kube-api-access-jbq5l\") pod \"rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7\" (UID: \"f739ec5f-a443-44c4-94d6-4077616af9c7\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" Nov 03 00:06:52.932148 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:52.932061 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" Nov 03 00:06:53.311462 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:53.311434 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7"] Nov 03 00:06:53.312469 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:06:53.312442 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf739ec5f_a443_44c4_94d6_4077616af9c7.slice/crio-54949425e63239de4a7c7be90e187b7a0ba6cbeb5d711fbb2aed7d533c186c9b WatchSource:0}: Error finding container 54949425e63239de4a7c7be90e187b7a0ba6cbeb5d711fbb2aed7d533c186c9b: Status 404 returned error can't find the container with id 54949425e63239de4a7c7be90e187b7a0ba6cbeb5d711fbb2aed7d533c186c9b Nov 03 00:06:53.821168 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:53.821108 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" event={"ID":"f739ec5f-a443-44c4-94d6-4077616af9c7","Type":"ContainerStarted","Data":"2134d49b6234a845a87089d8df8314545494c466bce21e88fd3722fa5725ae27"} Nov 03 00:06:53.821555 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:53.821178 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" event={"ID":"f739ec5f-a443-44c4-94d6-4077616af9c7","Type":"ContainerStarted","Data":"54949425e63239de4a7c7be90e187b7a0ba6cbeb5d711fbb2aed7d533c186c9b"} Nov 03 00:06:53.932309 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:53.932274 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" Nov 03 00:06:53.964603 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:53.964548 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" podStartSLOduration=1.9645330410000001 podStartE2EDuration="1.964533041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:06:53.962461853 +0000 UTC m=+4700.200959545" watchObservedRunningTime="2026-11-03 00:06:53.964533041 +0000 UTC m=+4700.203030699" Nov 03 00:06:54.073376 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:54.073276 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-96031a-bd8c6d645-lxnf7"] Nov 03 00:06:57.847860 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:06:57.847815 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:06:59.073629 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:06:59.073590 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-96031a-bd8c6d645-lxnf7" podUID="926c60cb-4000-4520-bde5-f31651560cc3" containerName="payload-processing" containerID="cri-o://67577f258006ded580aaf6706b757d2884522162c674fefbab9d7c2f2ec3af67" gracePeriod=25 Nov 03 00:07:00.746035 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:00.745998 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-96031a-bd8c6d645-lxnf7" podUID="926c60cb-4000-4520-bde5-f31651560cc3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:07:01.090971 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:01.090936 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-04713b77-7d67646d57-w65qh"] Nov 03 00:07:02.932300 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:02.932264 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" Nov 03 00:07:02.945410 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:02.945378 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" Nov 03 00:07:03.911292 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:03.911260 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" Nov 03 00:07:06.092154 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:06.092068 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-04713b77-7d67646d57-w65qh" podUID="d587d22d-31c7-4266-a7df-323f11fd4499" containerName="payload-processing" containerID="cri-o://276fb87e798d953642102aceba184530820981ba7d5fd09c5a6e11bd22bfbee6" gracePeriod=25 Nov 03 00:07:06.449070 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:06.449034 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-04713b77-7d67646d57-w65qh" Nov 03 00:07:06.479441 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:06.479406 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d587d22d-31c7-4266-a7df-323f11fd4499-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d587d22d-31c7-4266-a7df-323f11fd4499-plugins-config-volume\") pod \"d587d22d-31c7-4266-a7df-323f11fd4499\" (UID: \"d587d22d-31c7-4266-a7df-323f11fd4499\") " Nov 03 00:07:06.479874 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:06.479834 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d587d22d-31c7-4266-a7df-323f11fd4499-plugins-config-volume" pod "d587d22d-31c7-4266-a7df-323f11fd4499" (UID: "d587d22d-31c7-4266-a7df-323f11fd4499"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:07:06.480224 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:06.480192 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d587d22d-31c7-4266-a7df-323f11fd4499-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:06.585157 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:06.584690 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d587d22d-31c7-4266-a7df-323f11fd4499-kube-api-access-9fz6w\" (UniqueName: \"kubernetes.io/projected/d587d22d-31c7-4266-a7df-323f11fd4499-kube-api-access-9fz6w\") pod \"d587d22d-31c7-4266-a7df-323f11fd4499\" (UID: \"d587d22d-31c7-4266-a7df-323f11fd4499\") " Nov 03 00:07:06.598731 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:06.598675 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d587d22d-31c7-4266-a7df-323f11fd4499-kube-api-access-9fz6w" pod "d587d22d-31c7-4266-a7df-323f11fd4499" (UID: "d587d22d-31c7-4266-a7df-323f11fd4499"). InnerVolumeSpecName "kube-api-access-9fz6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:07:06.688859 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:06.688669 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fz6w\" (UniqueName: \"kubernetes.io/projected/d587d22d-31c7-4266-a7df-323f11fd4499-kube-api-access-9fz6w\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:06.928267 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:06.928225 5727 generic.go:356] "Generic (PLEG): container finished" podID="d587d22d-31c7-4266-a7df-323f11fd4499" containerID="276fb87e798d953642102aceba184530820981ba7d5fd09c5a6e11bd22bfbee6" exitCode=0 Nov 03 00:07:06.928463 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:06.928364 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-04713b77-7d67646d57-w65qh" event={"ID":"d587d22d-31c7-4266-a7df-323f11fd4499","Type":"ContainerDied","Data":"276fb87e798d953642102aceba184530820981ba7d5fd09c5a6e11bd22bfbee6"} Nov 03 00:07:06.928463 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:06.928416 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-04713b77-7d67646d57-w65qh" event={"ID":"d587d22d-31c7-4266-a7df-323f11fd4499","Type":"ContainerDied","Data":"909d1f63f951c76e8f9e504cdc5d7c3de6a3d9d27394f9775131c74462b9efd3"} Nov 03 00:07:06.928463 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:06.928444 5727 scope.go:122] "RemoveContainer" containerID="276fb87e798d953642102aceba184530820981ba7d5fd09c5a6e11bd22bfbee6" Nov 03 00:07:06.928628 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:06.928611 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-04713b77-7d67646d57-w65qh" Nov 03 00:07:06.949177 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:06.948420 5727 scope.go:122] "RemoveContainer" containerID="276fb87e798d953642102aceba184530820981ba7d5fd09c5a6e11bd22bfbee6" Nov 03 00:07:06.949177 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:07:06.948819 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"276fb87e798d953642102aceba184530820981ba7d5fd09c5a6e11bd22bfbee6\": container with ID starting with 276fb87e798d953642102aceba184530820981ba7d5fd09c5a6e11bd22bfbee6 not found: ID does not exist" containerID="276fb87e798d953642102aceba184530820981ba7d5fd09c5a6e11bd22bfbee6" Nov 03 00:07:06.949177 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:06.948856 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"276fb87e798d953642102aceba184530820981ba7d5fd09c5a6e11bd22bfbee6"} err="failed to get container status \"276fb87e798d953642102aceba184530820981ba7d5fd09c5a6e11bd22bfbee6\": rpc error: code = NotFound desc = could not find container \"276fb87e798d953642102aceba184530820981ba7d5fd09c5a6e11bd22bfbee6\": container with ID starting with 276fb87e798d953642102aceba184530820981ba7d5fd09c5a6e11bd22bfbee6 not found: ID does not exist" Nov 03 00:07:07.177340 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:07.177114 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-04713b77-7d67646d57-w65qh"] Nov 03 00:07:07.198237 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:07.194690 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-04713b77-7d67646d57-w65qh"] Nov 03 00:07:07.885979 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:07.885947 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d587d22d-31c7-4266-a7df-323f11fd4499" path="/var/lib/kubelet/pods/d587d22d-31c7-4266-a7df-323f11fd4499/volumes" Nov 03 00:07:10.746089 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:10.746048 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-96031a-bd8c6d645-lxnf7" podUID="926c60cb-4000-4520-bde5-f31651560cc3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:07:19.358887 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:19.358860 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-96031a-bd8c6d645-lxnf7" Nov 03 00:07:19.451205 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:19.451174 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/926c60cb-4000-4520-bde5-f31651560cc3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/926c60cb-4000-4520-bde5-f31651560cc3-plugins-config-volume\") pod \"926c60cb-4000-4520-bde5-f31651560cc3\" (UID: \"926c60cb-4000-4520-bde5-f31651560cc3\") " Nov 03 00:07:19.451401 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:19.451280 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/926c60cb-4000-4520-bde5-f31651560cc3-kube-api-access-vbwwt\" (UniqueName: \"kubernetes.io/projected/926c60cb-4000-4520-bde5-f31651560cc3-kube-api-access-vbwwt\") pod \"926c60cb-4000-4520-bde5-f31651560cc3\" (UID: \"926c60cb-4000-4520-bde5-f31651560cc3\") " Nov 03 00:07:19.451586 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:19.451561 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/926c60cb-4000-4520-bde5-f31651560cc3-plugins-config-volume" pod "926c60cb-4000-4520-bde5-f31651560cc3" (UID: "926c60cb-4000-4520-bde5-f31651560cc3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:07:19.451798 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:19.451779 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/926c60cb-4000-4520-bde5-f31651560cc3-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:19.453617 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:19.453587 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/926c60cb-4000-4520-bde5-f31651560cc3-kube-api-access-vbwwt" pod "926c60cb-4000-4520-bde5-f31651560cc3" (UID: "926c60cb-4000-4520-bde5-f31651560cc3"). InnerVolumeSpecName "kube-api-access-vbwwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:07:19.552681 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:19.552625 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbwwt\" (UniqueName: \"kubernetes.io/projected/926c60cb-4000-4520-bde5-f31651560cc3-kube-api-access-vbwwt\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:20.034664 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:20.034625 5727 generic.go:356] "Generic (PLEG): container finished" podID="926c60cb-4000-4520-bde5-f31651560cc3" containerID="67577f258006ded580aaf6706b757d2884522162c674fefbab9d7c2f2ec3af67" exitCode=0 Nov 03 00:07:20.034816 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:20.034693 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-96031a-bd8c6d645-lxnf7" Nov 03 00:07:20.034816 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:20.034712 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-96031a-bd8c6d645-lxnf7" event={"ID":"926c60cb-4000-4520-bde5-f31651560cc3","Type":"ContainerDied","Data":"67577f258006ded580aaf6706b757d2884522162c674fefbab9d7c2f2ec3af67"} Nov 03 00:07:20.034816 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:20.034766 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-96031a-bd8c6d645-lxnf7" event={"ID":"926c60cb-4000-4520-bde5-f31651560cc3","Type":"ContainerDied","Data":"d334370fb63857a8e649e6575fa4419eb9483969366a9b373fbe44256a76220c"} Nov 03 00:07:20.034816 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:20.034787 5727 scope.go:122] "RemoveContainer" containerID="67577f258006ded580aaf6706b757d2884522162c674fefbab9d7c2f2ec3af67" Nov 03 00:07:20.045291 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:20.045258 5727 scope.go:122] "RemoveContainer" containerID="67577f258006ded580aaf6706b757d2884522162c674fefbab9d7c2f2ec3af67" Nov 03 00:07:20.045542 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:07:20.045521 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67577f258006ded580aaf6706b757d2884522162c674fefbab9d7c2f2ec3af67\": container with ID starting with 67577f258006ded580aaf6706b757d2884522162c674fefbab9d7c2f2ec3af67 not found: ID does not exist" containerID="67577f258006ded580aaf6706b757d2884522162c674fefbab9d7c2f2ec3af67" Nov 03 00:07:20.045593 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:20.045554 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67577f258006ded580aaf6706b757d2884522162c674fefbab9d7c2f2ec3af67"} err="failed to get container status \"67577f258006ded580aaf6706b757d2884522162c674fefbab9d7c2f2ec3af67\": rpc error: code = NotFound desc = could not find container \"67577f258006ded580aaf6706b757d2884522162c674fefbab9d7c2f2ec3af67\": container with ID starting with 67577f258006ded580aaf6706b757d2884522162c674fefbab9d7c2f2ec3af67 not found: ID does not exist" Nov 03 00:07:20.198570 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:20.198487 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-96031a-bd8c6d645-lxnf7"] Nov 03 00:07:20.204304 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:20.204274 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-96031a-bd8c6d645-lxnf7"] Nov 03 00:07:21.773916 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:21.773877 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr"] Nov 03 00:07:21.774418 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:21.774193 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr" podUID="3b693031-94ce-4930-b763-6455d23bfea8" containerName="maas-api" containerID="cri-o://f7222ef1e8bae8fc15a535b083c5497f8b73bac888ffe57d0d8063c15ab45173" gracePeriod=30 Nov 03 00:07:21.885813 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:21.885777 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="926c60cb-4000-4520-bde5-f31651560cc3" path="/var/lib/kubelet/pods/926c60cb-4000-4520-bde5-f31651560cc3/volumes" Nov 03 00:07:22.054635 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:22.054604 5727 generic.go:356] "Generic (PLEG): container finished" podID="3b693031-94ce-4930-b763-6455d23bfea8" containerID="f7222ef1e8bae8fc15a535b083c5497f8b73bac888ffe57d0d8063c15ab45173" exitCode=0 Nov 03 00:07:22.054759 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:22.054674 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr" event={"ID":"3b693031-94ce-4930-b763-6455d23bfea8","Type":"ContainerDied","Data":"f7222ef1e8bae8fc15a535b083c5497f8b73bac888ffe57d0d8063c15ab45173"} Nov 03 00:07:22.054759 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:22.054736 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr" event={"ID":"3b693031-94ce-4930-b763-6455d23bfea8","Type":"ContainerDied","Data":"c87d291b0a8bfacabd9e6ec2dae2d5875899bde3c52e51dfbb3e2d406a079b55"} Nov 03 00:07:22.054845 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:22.054766 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c87d291b0a8bfacabd9e6ec2dae2d5875899bde3c52e51dfbb3e2d406a079b55" Nov 03 00:07:22.073591 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:22.073568 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr" Nov 03 00:07:22.194254 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:22.194224 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b693031-94ce-4930-b763-6455d23bfea8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b693031-94ce-4930-b763-6455d23bfea8-maas-api-tls\") pod \"3b693031-94ce-4930-b763-6455d23bfea8\" (UID: \"3b693031-94ce-4930-b763-6455d23bfea8\") " Nov 03 00:07:22.194421 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:22.194268 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b693031-94ce-4930-b763-6455d23bfea8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b693031-94ce-4930-b763-6455d23bfea8-openshift-service-ca\") pod \"3b693031-94ce-4930-b763-6455d23bfea8\" (UID: \"3b693031-94ce-4930-b763-6455d23bfea8\") " Nov 03 00:07:22.194421 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:22.194296 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b693031-94ce-4930-b763-6455d23bfea8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b693031-94ce-4930-b763-6455d23bfea8-metrics-certs\") pod \"3b693031-94ce-4930-b763-6455d23bfea8\" (UID: \"3b693031-94ce-4930-b763-6455d23bfea8\") " Nov 03 00:07:22.194557 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:22.194474 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b693031-94ce-4930-b763-6455d23bfea8-kube-api-access-4kgjw\" (UniqueName: \"kubernetes.io/projected/3b693031-94ce-4930-b763-6455d23bfea8-kube-api-access-4kgjw\") pod \"3b693031-94ce-4930-b763-6455d23bfea8\" (UID: \"3b693031-94ce-4930-b763-6455d23bfea8\") " Nov 03 00:07:22.194663 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:22.194636 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b693031-94ce-4930-b763-6455d23bfea8-openshift-service-ca" pod "3b693031-94ce-4930-b763-6455d23bfea8" (UID: "3b693031-94ce-4930-b763-6455d23bfea8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:07:22.194916 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:22.194897 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b693031-94ce-4930-b763-6455d23bfea8-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:22.197098 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:22.197069 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b693031-94ce-4930-b763-6455d23bfea8-maas-api-tls" pod "3b693031-94ce-4930-b763-6455d23bfea8" (UID: "3b693031-94ce-4930-b763-6455d23bfea8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:07:22.197254 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:22.197208 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b693031-94ce-4930-b763-6455d23bfea8-kube-api-access-4kgjw" pod "3b693031-94ce-4930-b763-6455d23bfea8" (UID: "3b693031-94ce-4930-b763-6455d23bfea8"). InnerVolumeSpecName "kube-api-access-4kgjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:07:22.197535 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:22.197512 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b693031-94ce-4930-b763-6455d23bfea8-metrics-certs" pod "3b693031-94ce-4930-b763-6455d23bfea8" (UID: "3b693031-94ce-4930-b763-6455d23bfea8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:07:22.295996 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:22.295917 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kgjw\" (UniqueName: \"kubernetes.io/projected/3b693031-94ce-4930-b763-6455d23bfea8-kube-api-access-4kgjw\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:22.295996 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:22.295946 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b693031-94ce-4930-b763-6455d23bfea8-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:22.295996 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:22.295959 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b693031-94ce-4930-b763-6455d23bfea8-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:23.061572 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:23.061538 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr" Nov 03 00:07:23.225852 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:23.225807 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr"] Nov 03 00:07:23.276879 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:23.276834 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-04713b77-8bdf95d9-jdrzr"] Nov 03 00:07:23.899657 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:23.899624 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b693031-94ce-4930-b763-6455d23bfea8" path="/var/lib/kubelet/pods/3b693031-94ce-4930-b763-6455d23bfea8/volumes" Nov 03 00:07:27.847643 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:07:27.847609 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:07:57.846196 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:07:57.846161 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:07:58.947870 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:58.947840 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7"] Nov 03 00:07:58.948265 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:07:58.948037 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" podUID="61ae60a1-87a0-42bf-9342-0420cd1cbce7" containerName="istio-proxy" containerID="cri-o://0f2ce3aae9ac1f1c50751326a8dfdb4d14dbfffefecb0ab82ad132739c82c9a0" gracePeriod=30 Nov 03 00:08:00.176332 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:00.176294 5727 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:08:00.176729 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:00.176352 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" podUID="61ae60a1-87a0-42bf-9342-0420cd1cbce7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:08:04.221174 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.221084 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 03 00:08:04.281572 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.281545 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istio-envoy\") pod \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " Nov 03 00:08:04.281739 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.281607 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-workload-socket\") pod \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " Nov 03 00:08:04.281739 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.281661 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istiod-ca-cert\") pod \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " Nov 03 00:08:04.281739 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.281691 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istio-token\" (UniqueName: \"kubernetes.io/projected/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istio-token\") pod \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " Nov 03 00:08:04.281739 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.281724 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istio-podinfo\") pod \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " Nov 03 00:08:04.281988 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.281784 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-workload-certs\") pod \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " Nov 03 00:08:04.281988 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.281880 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istio-data\") pod \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " Nov 03 00:08:04.281988 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.281916 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61ae60a1-87a0-42bf-9342-0420cd1cbce7-kube-api-access-k7psl\" (UniqueName: \"kubernetes.io/projected/61ae60a1-87a0-42bf-9342-0420cd1cbce7-kube-api-access-k7psl\") pod \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " Nov 03 00:08:04.281988 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.281951 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-credential-socket\") pod \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\" (UID: \"61ae60a1-87a0-42bf-9342-0420cd1cbce7\") " Nov 03 00:08:04.282269 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.281960 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-workload-socket" pod "61ae60a1-87a0-42bf-9342-0420cd1cbce7" (UID: "61ae60a1-87a0-42bf-9342-0420cd1cbce7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:04.282269 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.282245 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istio-data" pod "61ae60a1-87a0-42bf-9342-0420cd1cbce7" (UID: "61ae60a1-87a0-42bf-9342-0420cd1cbce7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:04.282269 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.282169 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istiod-ca-cert" pod "61ae60a1-87a0-42bf-9342-0420cd1cbce7" (UID: "61ae60a1-87a0-42bf-9342-0420cd1cbce7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:08:04.282508 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.282397 5727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istiod-ca-cert\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:04.282508 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.282414 5727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istio-data\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:04.282508 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.282428 5727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-workload-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:04.282508 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.282448 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-credential-socket" pod "61ae60a1-87a0-42bf-9342-0420cd1cbce7" (UID: "61ae60a1-87a0-42bf-9342-0420cd1cbce7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:04.282508 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.282466 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-workload-certs" pod "61ae60a1-87a0-42bf-9342-0420cd1cbce7" (UID: "61ae60a1-87a0-42bf-9342-0420cd1cbce7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:04.284673 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.284634 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61ae60a1-87a0-42bf-9342-0420cd1cbce7-kube-api-access-k7psl" pod "61ae60a1-87a0-42bf-9342-0420cd1cbce7" (UID: "61ae60a1-87a0-42bf-9342-0420cd1cbce7"). InnerVolumeSpecName "kube-api-access-k7psl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:08:04.284887 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.284858 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istio-envoy" pod "61ae60a1-87a0-42bf-9342-0420cd1cbce7" (UID: "61ae60a1-87a0-42bf-9342-0420cd1cbce7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:04.285019 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.284924 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istio-podinfo" pod "61ae60a1-87a0-42bf-9342-0420cd1cbce7" (UID: "61ae60a1-87a0-42bf-9342-0420cd1cbce7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:08:04.285019 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.284965 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istio-token" pod "61ae60a1-87a0-42bf-9342-0420cd1cbce7" (UID: "61ae60a1-87a0-42bf-9342-0420cd1cbce7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:08:04.378722 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.378684 5727 generic.go:356] "Generic (PLEG): container finished" podID="61ae60a1-87a0-42bf-9342-0420cd1cbce7" containerID="0f2ce3aae9ac1f1c50751326a8dfdb4d14dbfffefecb0ab82ad132739c82c9a0" exitCode=0 Nov 03 00:08:04.378928 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.378752 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" Nov 03 00:08:04.378928 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.378842 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" event={"ID":"61ae60a1-87a0-42bf-9342-0420cd1cbce7","Type":"ContainerDied","Data":"0f2ce3aae9ac1f1c50751326a8dfdb4d14dbfffefecb0ab82ad132739c82c9a0"} Nov 03 00:08:04.378928 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.378891 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7" event={"ID":"61ae60a1-87a0-42bf-9342-0420cd1cbce7","Type":"ContainerDied","Data":"0398ead25e66aa40304792c015bff44cef9875165f3c4814ba0229735530c435"} Nov 03 00:08:04.378928 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.378920 5727 scope.go:122] "RemoveContainer" containerID="0f2ce3aae9ac1f1c50751326a8dfdb4d14dbfffefecb0ab82ad132739c82c9a0" Nov 03 00:08:04.383305 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.383279 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7psl\" (UniqueName: \"kubernetes.io/projected/61ae60a1-87a0-42bf-9342-0420cd1cbce7-kube-api-access-k7psl\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:04.383448 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.383310 5727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-credential-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:04.383448 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.383326 5727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istio-envoy\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:04.383448 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.383351 5727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istio-token\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:04.383448 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.383366 5727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61ae60a1-87a0-42bf-9342-0420cd1cbce7-istio-podinfo\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:04.383448 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.383379 5727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61ae60a1-87a0-42bf-9342-0420cd1cbce7-workload-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:04.395240 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.395158 5727 scope.go:122] "RemoveContainer" containerID="0f2ce3aae9ac1f1c50751326a8dfdb4d14dbfffefecb0ab82ad132739c82c9a0" Nov 03 00:08:04.396056 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:08:04.395564 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f2ce3aae9ac1f1c50751326a8dfdb4d14dbfffefecb0ab82ad132739c82c9a0\": container with ID starting with 0f2ce3aae9ac1f1c50751326a8dfdb4d14dbfffefecb0ab82ad132739c82c9a0 not found: ID does not exist" containerID="0f2ce3aae9ac1f1c50751326a8dfdb4d14dbfffefecb0ab82ad132739c82c9a0" Nov 03 00:08:04.396056 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.395601 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f2ce3aae9ac1f1c50751326a8dfdb4d14dbfffefecb0ab82ad132739c82c9a0"} err="failed to get container status \"0f2ce3aae9ac1f1c50751326a8dfdb4d14dbfffefecb0ab82ad132739c82c9a0\": rpc error: code = NotFound desc = could not find container \"0f2ce3aae9ac1f1c50751326a8dfdb4d14dbfffefecb0ab82ad132739c82c9a0\": container with ID starting with 0f2ce3aae9ac1f1c50751326a8dfdb4d14dbfffefecb0ab82ad132739c82c9a0 not found: ID does not exist" Nov 03 00:08:04.505279 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.505184 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7"] Nov 03 00:08:04.510621 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:04.510596 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-fb6s7"] Nov 03 00:08:05.888079 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:05.888039 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61ae60a1-87a0-42bf-9342-0420cd1cbce7" path="/var/lib/kubelet/pods/61ae60a1-87a0-42bf-9342-0420cd1cbce7/volumes" Nov 03 00:08:27.845748 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:08:27.845709 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:08:34.968538 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:34.968504 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 03 00:08:34.985948 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:34.985923 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 03 00:08:35.072072 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:35.072042 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 03 00:08:35.094661 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:08:35.094631 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 03 00:08:57.853426 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:08:57.853387 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:09:27.846032 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:09:27.845978 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:09:36.279403 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:09:36.279371 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz"] Nov 03 00:09:51.301431 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:09:51.301334 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" podUID="a9048ee3-8ca0-4723-9009-f6033427b829" containerName="main" containerID="cri-o://44898f25bc176ffdbebb68b06f8174bbe03c58f8fb1a2a25edaee6a93bf61943" gracePeriod=45 Nov 03 00:09:51.608356 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:09:51.608330 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" Nov 03 00:09:51.733675 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:09:51.733632 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9048ee3-8ca0-4723-9009-f6033427b829-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9048ee3-8ca0-4723-9009-f6033427b829-model-cache\") pod \"a9048ee3-8ca0-4723-9009-f6033427b829\" (UID: \"a9048ee3-8ca0-4723-9009-f6033427b829\") " Nov 03 00:09:51.733861 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:09:51.733808 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9048ee3-8ca0-4723-9009-f6033427b829-tls-certs\" (UniqueName: \"kubernetes.io/secret/a9048ee3-8ca0-4723-9009-f6033427b829-tls-certs\") pod \"a9048ee3-8ca0-4723-9009-f6033427b829\" (UID: \"a9048ee3-8ca0-4723-9009-f6033427b829\") " Nov 03 00:09:51.733973 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:09:51.733954 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9048ee3-8ca0-4723-9009-f6033427b829-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9048ee3-8ca0-4723-9009-f6033427b829-tmp-dir\") pod \"a9048ee3-8ca0-4723-9009-f6033427b829\" (UID: \"a9048ee3-8ca0-4723-9009-f6033427b829\") " Nov 03 00:09:51.734029 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:09:51.734012 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9048ee3-8ca0-4723-9009-f6033427b829-kube-api-access-kj6dj\" (UniqueName: \"kubernetes.io/projected/a9048ee3-8ca0-4723-9009-f6033427b829-kube-api-access-kj6dj\") pod \"a9048ee3-8ca0-4723-9009-f6033427b829\" (UID: \"a9048ee3-8ca0-4723-9009-f6033427b829\") " Nov 03 00:09:51.734088 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:09:51.734072 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9048ee3-8ca0-4723-9009-f6033427b829-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9048ee3-8ca0-4723-9009-f6033427b829-dshm\") pod \"a9048ee3-8ca0-4723-9009-f6033427b829\" (UID: \"a9048ee3-8ca0-4723-9009-f6033427b829\") " Nov 03 00:09:51.734174 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:09:51.734122 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9048ee3-8ca0-4723-9009-f6033427b829-home\" (UniqueName: \"kubernetes.io/empty-dir/a9048ee3-8ca0-4723-9009-f6033427b829-home\") pod \"a9048ee3-8ca0-4723-9009-f6033427b829\" (UID: \"a9048ee3-8ca0-4723-9009-f6033427b829\") " Nov 03 00:09:51.735502 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:09:51.735445 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9048ee3-8ca0-4723-9009-f6033427b829-home" pod "a9048ee3-8ca0-4723-9009-f6033427b829" (UID: "a9048ee3-8ca0-4723-9009-f6033427b829"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:09:51.735709 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:09:51.735685 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9048ee3-8ca0-4723-9009-f6033427b829-model-cache" pod "a9048ee3-8ca0-4723-9009-f6033427b829" (UID: "a9048ee3-8ca0-4723-9009-f6033427b829"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:09:51.737404 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:09:51.737234 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9048ee3-8ca0-4723-9009-f6033427b829-tmp-dir" pod "a9048ee3-8ca0-4723-9009-f6033427b829" (UID: "a9048ee3-8ca0-4723-9009-f6033427b829"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:09:51.739202 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:09:51.738108 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9048ee3-8ca0-4723-9009-f6033427b829-kube-api-access-kj6dj" pod "a9048ee3-8ca0-4723-9009-f6033427b829" (UID: "a9048ee3-8ca0-4723-9009-f6033427b829"). InnerVolumeSpecName "kube-api-access-kj6dj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:09:51.741202 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:09:51.739995 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9048ee3-8ca0-4723-9009-f6033427b829-dshm" pod "a9048ee3-8ca0-4723-9009-f6033427b829" (UID: "a9048ee3-8ca0-4723-9009-f6033427b829"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:09:51.742913 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:09:51.742871 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9048ee3-8ca0-4723-9009-f6033427b829-tls-certs" pod "a9048ee3-8ca0-4723-9009-f6033427b829" (UID: "a9048ee3-8ca0-4723-9009-f6033427b829"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:09:51.836155 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:09:51.836050 5727 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9048ee3-8ca0-4723-9009-f6033427b829-home\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:51.836155 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:09:51.836096 5727 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9048ee3-8ca0-4723-9009-f6033427b829-model-cache\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:51.836155 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:09:51.836113 5727 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9048ee3-8ca0-4723-9009-f6033427b829-tls-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:51.836507 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:09:51.836160 5727 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9048ee3-8ca0-4723-9009-f6033427b829-tmp-dir\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:51.836507 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:09:51.836178 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj6dj\" (UniqueName: \"kubernetes.io/projected/a9048ee3-8ca0-4723-9009-f6033427b829-kube-api-access-kj6dj\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:51.836507 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:09:51.836193 5727 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9048ee3-8ca0-4723-9009-f6033427b829-dshm\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:52.184481 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:09:52.184389 5727 generic.go:356] "Generic (PLEG): container finished" podID="a9048ee3-8ca0-4723-9009-f6033427b829" containerID="44898f25bc176ffdbebb68b06f8174bbe03c58f8fb1a2a25edaee6a93bf61943" exitCode=0 Nov 03 00:09:52.184481 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:09:52.184458 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" Nov 03 00:09:52.184746 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:09:52.184486 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" event={"ID":"a9048ee3-8ca0-4723-9009-f6033427b829","Type":"ContainerDied","Data":"44898f25bc176ffdbebb68b06f8174bbe03c58f8fb1a2a25edaee6a93bf61943"} Nov 03 00:09:52.184746 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:09:52.184548 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz" event={"ID":"a9048ee3-8ca0-4723-9009-f6033427b829","Type":"ContainerDied","Data":"1041a79ae67fa18c2316bde721751574b07400038c5a26809cac3c62114293c1"} Nov 03 00:09:52.184746 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:09:52.184574 5727 scope.go:122] "RemoveContainer" containerID="44898f25bc176ffdbebb68b06f8174bbe03c58f8fb1a2a25edaee6a93bf61943" Nov 03 00:09:52.195102 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:09:52.195081 5727 scope.go:122] "RemoveContainer" containerID="44898f25bc176ffdbebb68b06f8174bbe03c58f8fb1a2a25edaee6a93bf61943" Nov 03 00:09:52.195418 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:09:52.195392 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44898f25bc176ffdbebb68b06f8174bbe03c58f8fb1a2a25edaee6a93bf61943\": container with ID starting with 44898f25bc176ffdbebb68b06f8174bbe03c58f8fb1a2a25edaee6a93bf61943 not found: ID does not exist" containerID="44898f25bc176ffdbebb68b06f8174bbe03c58f8fb1a2a25edaee6a93bf61943" Nov 03 00:09:52.195551 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:09:52.195425 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44898f25bc176ffdbebb68b06f8174bbe03c58f8fb1a2a25edaee6a93bf61943"} err="failed to get container status \"44898f25bc176ffdbebb68b06f8174bbe03c58f8fb1a2a25edaee6a93bf61943\": rpc error: code = NotFound desc = could not find container \"44898f25bc176ffdbebb68b06f8174bbe03c58f8fb1a2a25edaee6a93bf61943\": container with ID starting with 44898f25bc176ffdbebb68b06f8174bbe03c58f8fb1a2a25edaee6a93bf61943 not found: ID does not exist" Nov 03 00:09:52.349288 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:09:52.349255 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz"] Nov 03 00:09:52.356808 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:09:52.356774 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-637fdb/premium-w4-637fdb-kserve-79868cf966-dklxz"] Nov 03 00:09:53.883640 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:09:53.883606 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9048ee3-8ca0-4723-9009-f6033427b829" path="/var/lib/kubelet/pods/a9048ee3-8ca0-4723-9009-f6033427b829/volumes" Nov 03 00:09:57.849274 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:09:57.849234 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:10:27.845623 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:10:27.845581 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:10:31.121666 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:31.121633 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-637fdb-97f6995c-szpck"] Nov 03 00:10:31.122073 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:31.121874 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-637fdb-97f6995c-szpck" podUID="5eff97c2-fd05-445f-bc27-2d5f56df0f09" containerName="maas-api" containerID="cri-o://a8d9bf24fe0afb3ae71297954975aaa18a675ab974e76dc3c602a33907d0e89a" gracePeriod=30 Nov 03 00:10:31.391714 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:31.391644 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-637fdb-97f6995c-szpck" Nov 03 00:10:31.477997 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:31.477958 5727 generic.go:356] "Generic (PLEG): container finished" podID="5eff97c2-fd05-445f-bc27-2d5f56df0f09" containerID="a8d9bf24fe0afb3ae71297954975aaa18a675ab974e76dc3c602a33907d0e89a" exitCode=0 Nov 03 00:10:31.478227 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:31.478011 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-637fdb-97f6995c-szpck" Nov 03 00:10:31.478227 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:31.478046 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-637fdb-97f6995c-szpck" event={"ID":"5eff97c2-fd05-445f-bc27-2d5f56df0f09","Type":"ContainerDied","Data":"a8d9bf24fe0afb3ae71297954975aaa18a675ab974e76dc3c602a33907d0e89a"} Nov 03 00:10:31.478227 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:31.478110 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-637fdb-97f6995c-szpck" event={"ID":"5eff97c2-fd05-445f-bc27-2d5f56df0f09","Type":"ContainerDied","Data":"0f7aaad57c8d8a9b01dec90471d081d326a69ba07711025c733d0526333d2407"} Nov 03 00:10:31.478227 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:31.478148 5727 scope.go:122] "RemoveContainer" containerID="a8d9bf24fe0afb3ae71297954975aaa18a675ab974e76dc3c602a33907d0e89a" Nov 03 00:10:31.489627 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:31.489605 5727 scope.go:122] "RemoveContainer" containerID="a8d9bf24fe0afb3ae71297954975aaa18a675ab974e76dc3c602a33907d0e89a" Nov 03 00:10:31.489981 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:10:31.489959 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8d9bf24fe0afb3ae71297954975aaa18a675ab974e76dc3c602a33907d0e89a\": container with ID starting with a8d9bf24fe0afb3ae71297954975aaa18a675ab974e76dc3c602a33907d0e89a not found: ID does not exist" containerID="a8d9bf24fe0afb3ae71297954975aaa18a675ab974e76dc3c602a33907d0e89a" Nov 03 00:10:31.490036 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:31.489996 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8d9bf24fe0afb3ae71297954975aaa18a675ab974e76dc3c602a33907d0e89a"} err="failed to get container status \"a8d9bf24fe0afb3ae71297954975aaa18a675ab974e76dc3c602a33907d0e89a\": rpc error: code = NotFound desc = could not find container \"a8d9bf24fe0afb3ae71297954975aaa18a675ab974e76dc3c602a33907d0e89a\": container with ID starting with a8d9bf24fe0afb3ae71297954975aaa18a675ab974e76dc3c602a33907d0e89a not found: ID does not exist" Nov 03 00:10:31.520151 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:31.520082 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5eff97c2-fd05-445f-bc27-2d5f56df0f09-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eff97c2-fd05-445f-bc27-2d5f56df0f09-metrics-certs\") pod \"5eff97c2-fd05-445f-bc27-2d5f56df0f09\" (UID: \"5eff97c2-fd05-445f-bc27-2d5f56df0f09\") " Nov 03 00:10:31.520492 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:31.520470 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5eff97c2-fd05-445f-bc27-2d5f56df0f09-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eff97c2-fd05-445f-bc27-2d5f56df0f09-maas-api-tls\") pod \"5eff97c2-fd05-445f-bc27-2d5f56df0f09\" (UID: \"5eff97c2-fd05-445f-bc27-2d5f56df0f09\") " Nov 03 00:10:31.520796 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:31.520779 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eff97c2-fd05-445f-bc27-2d5f56df0f09-kube-api-access-hqqmr\" (UniqueName: \"kubernetes.io/projected/5eff97c2-fd05-445f-bc27-2d5f56df0f09-kube-api-access-hqqmr\") pod \"5eff97c2-fd05-445f-bc27-2d5f56df0f09\" (UID: \"5eff97c2-fd05-445f-bc27-2d5f56df0f09\") " Nov 03 00:10:31.521002 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:31.520969 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eff97c2-fd05-445f-bc27-2d5f56df0f09-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eff97c2-fd05-445f-bc27-2d5f56df0f09-openshift-service-ca\") pod \"5eff97c2-fd05-445f-bc27-2d5f56df0f09\" (UID: \"5eff97c2-fd05-445f-bc27-2d5f56df0f09\") " Nov 03 00:10:31.522227 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:31.522200 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eff97c2-fd05-445f-bc27-2d5f56df0f09-openshift-service-ca" pod "5eff97c2-fd05-445f-bc27-2d5f56df0f09" (UID: "5eff97c2-fd05-445f-bc27-2d5f56df0f09"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:10:31.528774 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:31.525235 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eff97c2-fd05-445f-bc27-2d5f56df0f09-maas-api-tls" pod "5eff97c2-fd05-445f-bc27-2d5f56df0f09" (UID: "5eff97c2-fd05-445f-bc27-2d5f56df0f09"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:10:31.529283 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:31.529240 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eff97c2-fd05-445f-bc27-2d5f56df0f09-kube-api-access-hqqmr" pod "5eff97c2-fd05-445f-bc27-2d5f56df0f09" (UID: "5eff97c2-fd05-445f-bc27-2d5f56df0f09"). InnerVolumeSpecName "kube-api-access-hqqmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:10:31.536615 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:31.536581 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eff97c2-fd05-445f-bc27-2d5f56df0f09-metrics-certs" pod "5eff97c2-fd05-445f-bc27-2d5f56df0f09" (UID: "5eff97c2-fd05-445f-bc27-2d5f56df0f09"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:10:31.623968 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:31.623906 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eff97c2-fd05-445f-bc27-2d5f56df0f09-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:31.623968 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:31.623940 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eff97c2-fd05-445f-bc27-2d5f56df0f09-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:31.623968 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:31.623959 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqqmr\" (UniqueName: \"kubernetes.io/projected/5eff97c2-fd05-445f-bc27-2d5f56df0f09-kube-api-access-hqqmr\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:31.623968 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:31.623972 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eff97c2-fd05-445f-bc27-2d5f56df0f09-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:31.898539 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:31.898488 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-637fdb-97f6995c-szpck"] Nov 03 00:10:31.903812 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:31.903783 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-637fdb-97f6995c-szpck"] Nov 03 00:10:33.884324 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:33.884288 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eff97c2-fd05-445f-bc27-2d5f56df0f09" path="/var/lib/kubelet/pods/5eff97c2-fd05-445f-bc27-2d5f56df0f09/volumes" Nov 03 00:10:35.741711 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:35.741678 5727 scope.go:122] "RemoveContainer" containerID="f7222ef1e8bae8fc15a535b083c5497f8b73bac888ffe57d0d8063c15ab45173" Nov 03 00:10:40.489687 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:40.489641 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m"] Nov 03 00:10:40.490223 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:40.489905 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" podUID="fe18fd85-9a72-4868-8763-038ea4a0a54a" containerName="istio-proxy" containerID="cri-o://3941125a1b0a69e35c41042d94b10ef0841b5daec3ef5e9fde4d821d4c6c9bae" gracePeriod=30 Nov 03 00:10:45.750716 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:45.750691 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 03 00:10:45.888554 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:45.888522 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-credential-socket\") pod \"fe18fd85-9a72-4868-8763-038ea4a0a54a\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " Nov 03 00:10:45.888711 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:45.888621 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-istio-envoy\") pod \"fe18fd85-9a72-4868-8763-038ea4a0a54a\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " Nov 03 00:10:45.888751 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:45.888716 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe18fd85-9a72-4868-8763-038ea4a0a54a-kube-api-access-7pjsd\" (UniqueName: \"kubernetes.io/projected/fe18fd85-9a72-4868-8763-038ea4a0a54a-kube-api-access-7pjsd\") pod \"fe18fd85-9a72-4868-8763-038ea4a0a54a\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " Nov 03 00:10:45.888807 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:45.888754 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe18fd85-9a72-4868-8763-038ea4a0a54a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe18fd85-9a72-4868-8763-038ea4a0a54a-istio-podinfo\") pod \"fe18fd85-9a72-4868-8763-038ea4a0a54a\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " Nov 03 00:10:45.888850 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:45.888812 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-credential-socket" pod "fe18fd85-9a72-4868-8763-038ea4a0a54a" (UID: "fe18fd85-9a72-4868-8763-038ea4a0a54a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:10:45.888907 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:45.888854 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe18fd85-9a72-4868-8763-038ea4a0a54a-istio-token\" (UniqueName: \"kubernetes.io/projected/fe18fd85-9a72-4868-8763-038ea4a0a54a-istio-token\") pod \"fe18fd85-9a72-4868-8763-038ea4a0a54a\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " Nov 03 00:10:45.888907 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:45.888873 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-istio-data\") pod \"fe18fd85-9a72-4868-8763-038ea4a0a54a\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " Nov 03 00:10:45.889028 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:45.888927 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-workload-socket\") pod \"fe18fd85-9a72-4868-8763-038ea4a0a54a\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " Nov 03 00:10:45.889028 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:45.888983 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe18fd85-9a72-4868-8763-038ea4a0a54a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe18fd85-9a72-4868-8763-038ea4a0a54a-istiod-ca-cert\") pod \"fe18fd85-9a72-4868-8763-038ea4a0a54a\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " Nov 03 00:10:45.889028 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:45.889014 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-workload-certs\") pod \"fe18fd85-9a72-4868-8763-038ea4a0a54a\" (UID: \"fe18fd85-9a72-4868-8763-038ea4a0a54a\") " Nov 03 00:10:45.889250 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:45.889174 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-istio-data" pod "fe18fd85-9a72-4868-8763-038ea4a0a54a" (UID: "fe18fd85-9a72-4868-8763-038ea4a0a54a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:10:45.889625 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:45.889395 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-workload-certs" pod "fe18fd85-9a72-4868-8763-038ea4a0a54a" (UID: "fe18fd85-9a72-4868-8763-038ea4a0a54a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:10:45.889747 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:45.889660 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe18fd85-9a72-4868-8763-038ea4a0a54a-istiod-ca-cert" pod "fe18fd85-9a72-4868-8763-038ea4a0a54a" (UID: "fe18fd85-9a72-4868-8763-038ea4a0a54a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:10:45.889747 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:45.889702 5727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-istio-data\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:45.889747 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:45.889725 5727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-credential-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:45.889747 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:45.889724 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-workload-socket" pod "fe18fd85-9a72-4868-8763-038ea4a0a54a" (UID: "fe18fd85-9a72-4868-8763-038ea4a0a54a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:10:45.891196 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:45.891173 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe18fd85-9a72-4868-8763-038ea4a0a54a-istio-token" pod "fe18fd85-9a72-4868-8763-038ea4a0a54a" (UID: "fe18fd85-9a72-4868-8763-038ea4a0a54a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:10:45.891196 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:45.891174 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe18fd85-9a72-4868-8763-038ea4a0a54a-istio-podinfo" pod "fe18fd85-9a72-4868-8763-038ea4a0a54a" (UID: "fe18fd85-9a72-4868-8763-038ea4a0a54a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:10:45.891364 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:45.891266 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe18fd85-9a72-4868-8763-038ea4a0a54a-kube-api-access-7pjsd" pod "fe18fd85-9a72-4868-8763-038ea4a0a54a" (UID: "fe18fd85-9a72-4868-8763-038ea4a0a54a"). InnerVolumeSpecName "kube-api-access-7pjsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:10:45.891364 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:45.891275 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-istio-envoy" pod "fe18fd85-9a72-4868-8763-038ea4a0a54a" (UID: "fe18fd85-9a72-4868-8763-038ea4a0a54a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:10:45.990463 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:45.990435 5727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-istio-envoy\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:45.990463 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:45.990463 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pjsd\" (UniqueName: \"kubernetes.io/projected/fe18fd85-9a72-4868-8763-038ea4a0a54a-kube-api-access-7pjsd\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:45.990664 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:45.990473 5727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe18fd85-9a72-4868-8763-038ea4a0a54a-istio-podinfo\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:45.990664 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:45.990482 5727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe18fd85-9a72-4868-8763-038ea4a0a54a-istio-token\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:45.990664 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:45.990490 5727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-workload-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:45.990664 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:45.990501 5727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe18fd85-9a72-4868-8763-038ea4a0a54a-istiod-ca-cert\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:45.990664 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:45.990509 5727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe18fd85-9a72-4868-8763-038ea4a0a54a-workload-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:46.591759 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:46.591722 5727 generic.go:356] "Generic (PLEG): container finished" podID="fe18fd85-9a72-4868-8763-038ea4a0a54a" containerID="3941125a1b0a69e35c41042d94b10ef0841b5daec3ef5e9fde4d821d4c6c9bae" exitCode=0 Nov 03 00:10:46.591965 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:46.591779 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" event={"ID":"fe18fd85-9a72-4868-8763-038ea4a0a54a","Type":"ContainerDied","Data":"3941125a1b0a69e35c41042d94b10ef0841b5daec3ef5e9fde4d821d4c6c9bae"} Nov 03 00:10:46.591965 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:46.591796 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" Nov 03 00:10:46.591965 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:46.591829 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m" event={"ID":"fe18fd85-9a72-4868-8763-038ea4a0a54a","Type":"ContainerDied","Data":"12cf1ebf8bcae65815763df6ebb73791f55166aa9ca8c0175b4c46692ad8e356"} Nov 03 00:10:46.591965 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:46.591855 5727 scope.go:122] "RemoveContainer" containerID="3941125a1b0a69e35c41042d94b10ef0841b5daec3ef5e9fde4d821d4c6c9bae" Nov 03 00:10:46.603808 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:46.603673 5727 scope.go:122] "RemoveContainer" containerID="3941125a1b0a69e35c41042d94b10ef0841b5daec3ef5e9fde4d821d4c6c9bae" Nov 03 00:10:46.604006 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:10:46.603982 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3941125a1b0a69e35c41042d94b10ef0841b5daec3ef5e9fde4d821d4c6c9bae\": container with ID starting with 3941125a1b0a69e35c41042d94b10ef0841b5daec3ef5e9fde4d821d4c6c9bae not found: ID does not exist" containerID="3941125a1b0a69e35c41042d94b10ef0841b5daec3ef5e9fde4d821d4c6c9bae" Nov 03 00:10:46.604053 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:46.604016 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3941125a1b0a69e35c41042d94b10ef0841b5daec3ef5e9fde4d821d4c6c9bae"} err="failed to get container status \"3941125a1b0a69e35c41042d94b10ef0841b5daec3ef5e9fde4d821d4c6c9bae\": rpc error: code = NotFound desc = could not find container \"3941125a1b0a69e35c41042d94b10ef0841b5daec3ef5e9fde4d821d4c6c9bae\": container with ID starting with 3941125a1b0a69e35c41042d94b10ef0841b5daec3ef5e9fde4d821d4c6c9bae not found: ID does not exist" Nov 03 00:10:46.806711 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:46.806665 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m"] Nov 03 00:10:46.810598 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:46.810572 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-sf89m"] Nov 03 00:10:47.883832 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:47.883797 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe18fd85-9a72-4868-8763-038ea4a0a54a" path="/var/lib/kubelet/pods/fe18fd85-9a72-4868-8763-038ea4a0a54a/volumes" Nov 03 00:10:51.520353 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:51.520316 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d"] Nov 03 00:10:51.520804 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:51.520539 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" podUID="94871baf-708f-4190-b685-87f0e8ece5d4" containerName="istio-proxy" containerID="cri-o://2f7dfe058b645fe827e29244e08cdffe022084d2ba00a74e0972f8a8886c6850" gracePeriod=30 Nov 03 00:10:51.572212 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:51.571156 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn"] Nov 03 00:10:51.572212 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:51.571508 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" podUID="71f62bea-0cbe-4c6d-9cd3-63893df09a9e" containerName="istio-proxy" containerID="cri-o://2330b5cfa7071fa6edcd2570bf3c36869c20f61c8671085aaaf7ef0616870735" gracePeriod=30 Nov 03 00:10:53.521237 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:53.521197 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-znv9d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:10:53.521667 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:53.521274 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" podUID="94871baf-708f-4190-b685-87f0e8ece5d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:10:54.130095 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:54.130044 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-svghn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:10:54.130294 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:54.130110 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" podUID="71f62bea-0cbe-4c6d-9cd3-63893df09a9e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:10:56.676850 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:56.676810 5727 generic.go:356] "Generic (PLEG): container finished" podID="94871baf-708f-4190-b685-87f0e8ece5d4" containerID="2f7dfe058b645fe827e29244e08cdffe022084d2ba00a74e0972f8a8886c6850" exitCode=0 Nov 03 00:10:56.677303 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:56.676957 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" event={"ID":"94871baf-708f-4190-b685-87f0e8ece5d4","Type":"ContainerDied","Data":"2f7dfe058b645fe827e29244e08cdffe022084d2ba00a74e0972f8a8886c6850"} Nov 03 00:10:56.895050 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:56.895023 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 03 00:10:56.898717 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:56.898697 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 03 00:10:56.910187 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:56.910161 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94871baf-708f-4190-b685-87f0e8ece5d4-kube-api-access-tj4x2\" (UniqueName: \"kubernetes.io/projected/94871baf-708f-4190-b685-87f0e8ece5d4-kube-api-access-tj4x2\") pod \"94871baf-708f-4190-b685-87f0e8ece5d4\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " Nov 03 00:10:56.910317 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:56.910211 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-credential-socket\") pod \"94871baf-708f-4190-b685-87f0e8ece5d4\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " Nov 03 00:10:56.910317 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:56.910269 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94871baf-708f-4190-b685-87f0e8ece5d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94871baf-708f-4190-b685-87f0e8ece5d4-istiod-ca-cert\") pod \"94871baf-708f-4190-b685-87f0e8ece5d4\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " Nov 03 00:10:56.910317 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:56.910296 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-istio-data\") pod \"94871baf-708f-4190-b685-87f0e8ece5d4\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " Nov 03 00:10:56.910317 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:56.910315 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94871baf-708f-4190-b685-87f0e8ece5d4-istio-token\" (UniqueName: \"kubernetes.io/projected/94871baf-708f-4190-b685-87f0e8ece5d4-istio-token\") pod \"94871baf-708f-4190-b685-87f0e8ece5d4\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " Nov 03 00:10:56.910555 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:56.910390 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-istio-envoy\") pod \"94871baf-708f-4190-b685-87f0e8ece5d4\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " Nov 03 00:10:56.910555 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:56.910491 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-workload-socket\") pod \"94871baf-708f-4190-b685-87f0e8ece5d4\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " Nov 03 00:10:56.910555 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:56.910545 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-workload-certs\") pod \"94871baf-708f-4190-b685-87f0e8ece5d4\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " Nov 03 00:10:56.910717 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:56.910555 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-credential-socket" pod "94871baf-708f-4190-b685-87f0e8ece5d4" (UID: "94871baf-708f-4190-b685-87f0e8ece5d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:10:56.910717 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:56.910586 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94871baf-708f-4190-b685-87f0e8ece5d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94871baf-708f-4190-b685-87f0e8ece5d4-istio-podinfo\") pod \"94871baf-708f-4190-b685-87f0e8ece5d4\" (UID: \"94871baf-708f-4190-b685-87f0e8ece5d4\") " Nov 03 00:10:56.910717 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:56.910614 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-istio-data" pod "94871baf-708f-4190-b685-87f0e8ece5d4" (UID: "94871baf-708f-4190-b685-87f0e8ece5d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:10:56.910884 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:56.910802 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-workload-socket" pod "94871baf-708f-4190-b685-87f0e8ece5d4" (UID: "94871baf-708f-4190-b685-87f0e8ece5d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:10:56.910884 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:56.910840 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-workload-certs" pod "94871baf-708f-4190-b685-87f0e8ece5d4" (UID: "94871baf-708f-4190-b685-87f0e8ece5d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:10:56.911186 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:56.911157 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94871baf-708f-4190-b685-87f0e8ece5d4-istiod-ca-cert" pod "94871baf-708f-4190-b685-87f0e8ece5d4" (UID: "94871baf-708f-4190-b685-87f0e8ece5d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:10:56.911300 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:56.911166 5727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-credential-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:56.911300 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:56.911214 5727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-istio-data\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:56.911300 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:56.911232 5727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-workload-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:56.911300 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:56.911249 5727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-workload-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:56.913352 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:56.913320 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94871baf-708f-4190-b685-87f0e8ece5d4-kube-api-access-tj4x2" pod "94871baf-708f-4190-b685-87f0e8ece5d4" (UID: "94871baf-708f-4190-b685-87f0e8ece5d4"). InnerVolumeSpecName "kube-api-access-tj4x2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:10:56.913352 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:56.913336 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-istio-envoy" pod "94871baf-708f-4190-b685-87f0e8ece5d4" (UID: "94871baf-708f-4190-b685-87f0e8ece5d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:10:56.913352 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:56.913338 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94871baf-708f-4190-b685-87f0e8ece5d4-istio-token" pod "94871baf-708f-4190-b685-87f0e8ece5d4" (UID: "94871baf-708f-4190-b685-87f0e8ece5d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:10:56.913515 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:56.913419 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94871baf-708f-4190-b685-87f0e8ece5d4-istio-podinfo" pod "94871baf-708f-4190-b685-87f0e8ece5d4" (UID: "94871baf-708f-4190-b685-87f0e8ece5d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:10:57.012668 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.012619 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istiod-ca-cert\") pod \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " Nov 03 00:10:57.012874 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.012728 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-workload-socket\") pod \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " Nov 03 00:10:57.012874 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.012755 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istio-envoy\") pod \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " Nov 03 00:10:57.012874 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.012772 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-credential-socket\") pod \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " Nov 03 00:10:57.012874 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.012811 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istio-data\") pod \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " Nov 03 00:10:57.012874 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.012875 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istio-podinfo\") pod \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " Nov 03 00:10:57.013214 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.012909 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-kube-api-access-js2bw\" (UniqueName: \"kubernetes.io/projected/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-kube-api-access-js2bw\") pod \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " Nov 03 00:10:57.013214 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.012991 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istio-token\" (UniqueName: \"kubernetes.io/projected/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istio-token\") pod \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " Nov 03 00:10:57.013214 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.013032 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-workload-certs\") pod \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\" (UID: \"71f62bea-0cbe-4c6d-9cd3-63893df09a9e\") " Nov 03 00:10:57.013214 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.013027 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istiod-ca-cert" pod "71f62bea-0cbe-4c6d-9cd3-63893df09a9e" (UID: "71f62bea-0cbe-4c6d-9cd3-63893df09a9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:10:57.013214 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.013079 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-workload-socket" pod "71f62bea-0cbe-4c6d-9cd3-63893df09a9e" (UID: "71f62bea-0cbe-4c6d-9cd3-63893df09a9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:10:57.013214 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.013104 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-credential-socket" pod "71f62bea-0cbe-4c6d-9cd3-63893df09a9e" (UID: "71f62bea-0cbe-4c6d-9cd3-63893df09a9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:10:57.013759 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.013213 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istio-data" pod "71f62bea-0cbe-4c6d-9cd3-63893df09a9e" (UID: "71f62bea-0cbe-4c6d-9cd3-63893df09a9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:10:57.013759 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.013425 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-workload-certs" pod "71f62bea-0cbe-4c6d-9cd3-63893df09a9e" (UID: "71f62bea-0cbe-4c6d-9cd3-63893df09a9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:10:57.013759 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.013555 5727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94871baf-708f-4190-b685-87f0e8ece5d4-istio-envoy\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:57.013759 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.013578 5727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istiod-ca-cert\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:57.013759 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.013594 5727 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-workload-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:57.014031 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.013627 5727 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-credential-socket\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:57.014031 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.013812 5727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94871baf-708f-4190-b685-87f0e8ece5d4-istio-podinfo\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:57.014031 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.013825 5727 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istio-data\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:57.014031 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.013842 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj4x2\" (UniqueName: \"kubernetes.io/projected/94871baf-708f-4190-b685-87f0e8ece5d4-kube-api-access-tj4x2\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:57.014031 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.013858 5727 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94871baf-708f-4190-b685-87f0e8ece5d4-istiod-ca-cert\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:57.014031 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.013873 5727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94871baf-708f-4190-b685-87f0e8ece5d4-istio-token\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:57.014031 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.013887 5727 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-workload-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:57.015421 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.015395 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-kube-api-access-js2bw" pod "71f62bea-0cbe-4c6d-9cd3-63893df09a9e" (UID: "71f62bea-0cbe-4c6d-9cd3-63893df09a9e"). InnerVolumeSpecName "kube-api-access-js2bw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:10:57.015421 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.015407 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istio-podinfo" pod "71f62bea-0cbe-4c6d-9cd3-63893df09a9e" (UID: "71f62bea-0cbe-4c6d-9cd3-63893df09a9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:10:57.015421 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.015413 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istio-envoy" pod "71f62bea-0cbe-4c6d-9cd3-63893df09a9e" (UID: "71f62bea-0cbe-4c6d-9cd3-63893df09a9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:10:57.015615 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.015431 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istio-token" pod "71f62bea-0cbe-4c6d-9cd3-63893df09a9e" (UID: "71f62bea-0cbe-4c6d-9cd3-63893df09a9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:10:57.115576 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.115465 5727 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istio-envoy\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:57.115576 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.115494 5727 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istio-podinfo\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:57.115576 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.115504 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js2bw\" (UniqueName: \"kubernetes.io/projected/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-kube-api-access-js2bw\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:57.115576 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.115521 5727 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71f62bea-0cbe-4c6d-9cd3-63893df09a9e-istio-token\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:57.686044 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.686009 5727 generic.go:356] "Generic (PLEG): container finished" podID="71f62bea-0cbe-4c6d-9cd3-63893df09a9e" containerID="2330b5cfa7071fa6edcd2570bf3c36869c20f61c8671085aaaf7ef0616870735" exitCode=0 Nov 03 00:10:57.686580 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.686063 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" Nov 03 00:10:57.686580 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.686091 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" event={"ID":"71f62bea-0cbe-4c6d-9cd3-63893df09a9e","Type":"ContainerDied","Data":"2330b5cfa7071fa6edcd2570bf3c36869c20f61c8671085aaaf7ef0616870735"} Nov 03 00:10:57.686580 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.686159 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn" event={"ID":"71f62bea-0cbe-4c6d-9cd3-63893df09a9e","Type":"ContainerDied","Data":"d64cf8bb0df2fb86aca6723f681f079fd813f77d775a186dc10ede90b6a56b0d"} Nov 03 00:10:57.686580 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.686185 5727 scope.go:122] "RemoveContainer" containerID="2330b5cfa7071fa6edcd2570bf3c36869c20f61c8671085aaaf7ef0616870735" Nov 03 00:10:57.688006 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.687808 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" Nov 03 00:10:57.688006 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.687833 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d" event={"ID":"94871baf-708f-4190-b685-87f0e8ece5d4","Type":"ContainerDied","Data":"f1de949bea52ba9b856b259f42dff08aab101cc9d0b15e5559613c5512a0183b"} Nov 03 00:10:57.706072 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.706054 5727 scope.go:122] "RemoveContainer" containerID="2330b5cfa7071fa6edcd2570bf3c36869c20f61c8671085aaaf7ef0616870735" Nov 03 00:10:57.706384 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:10:57.706367 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2330b5cfa7071fa6edcd2570bf3c36869c20f61c8671085aaaf7ef0616870735\": container with ID starting with 2330b5cfa7071fa6edcd2570bf3c36869c20f61c8671085aaaf7ef0616870735 not found: ID does not exist" containerID="2330b5cfa7071fa6edcd2570bf3c36869c20f61c8671085aaaf7ef0616870735" Nov 03 00:10:57.706464 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.706396 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2330b5cfa7071fa6edcd2570bf3c36869c20f61c8671085aaaf7ef0616870735"} err="failed to get container status \"2330b5cfa7071fa6edcd2570bf3c36869c20f61c8671085aaaf7ef0616870735\": rpc error: code = NotFound desc = could not find container \"2330b5cfa7071fa6edcd2570bf3c36869c20f61c8671085aaaf7ef0616870735\": container with ID starting with 2330b5cfa7071fa6edcd2570bf3c36869c20f61c8671085aaaf7ef0616870735 not found: ID does not exist" Nov 03 00:10:57.706464 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.706419 5727 scope.go:122] "RemoveContainer" containerID="2f7dfe058b645fe827e29244e08cdffe022084d2ba00a74e0972f8a8886c6850" Nov 03 00:10:57.847094 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:10:57.847057 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:10:57.902682 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.902647 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d"] Nov 03 00:10:57.951055 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:57.951017 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znv9d"] Nov 03 00:10:58.014970 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:58.014936 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn"] Nov 03 00:10:58.022389 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:58.022357 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-svghn"] Nov 03 00:10:59.883855 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:59.883812 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71f62bea-0cbe-4c6d-9cd3-63893df09a9e" path="/var/lib/kubelet/pods/71f62bea-0cbe-4c6d-9cd3-63893df09a9e/volumes" Nov 03 00:10:59.884496 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:10:59.884342 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94871baf-708f-4190-b685-87f0e8ece5d4" path="/var/lib/kubelet/pods/94871baf-708f-4190-b685-87f0e8ece5d4/volumes" Nov 03 00:11:27.846570 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:11:27.846532 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:11:28.824453 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:11:28.824419 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7"] Nov 03 00:11:43.845548 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:11:43.845508 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" podUID="f739ec5f-a443-44c4-94d6-4077616af9c7" containerName="main" containerID="cri-o://2134d49b6234a845a87089d8df8314545494c466bce21e88fd3722fa5725ae27" gracePeriod=45 Nov 03 00:11:43.900685 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:11:43.900654 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" podUID="f739ec5f-a443-44c4-94d6-4077616af9c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": dial tcp 10.132.0.195:8000: connect: connection refused" Nov 03 00:11:44.027642 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:11:44.027604 5727 generic.go:356] "Generic (PLEG): container finished" podID="f739ec5f-a443-44c4-94d6-4077616af9c7" containerID="2134d49b6234a845a87089d8df8314545494c466bce21e88fd3722fa5725ae27" exitCode=0 Nov 03 00:11:44.027846 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:11:44.027825 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" event={"ID":"f739ec5f-a443-44c4-94d6-4077616af9c7","Type":"ContainerDied","Data":"2134d49b6234a845a87089d8df8314545494c466bce21e88fd3722fa5725ae27"} Nov 03 00:11:44.120879 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:11:44.120853 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" Nov 03 00:11:44.240866 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:11:44.240833 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f739ec5f-a443-44c4-94d6-4077616af9c7-tls-certs\" (UniqueName: \"kubernetes.io/secret/f739ec5f-a443-44c4-94d6-4077616af9c7-tls-certs\") pod \"f739ec5f-a443-44c4-94d6-4077616af9c7\" (UID: \"f739ec5f-a443-44c4-94d6-4077616af9c7\") " Nov 03 00:11:44.241049 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:11:44.240894 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f739ec5f-a443-44c4-94d6-4077616af9c7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f739ec5f-a443-44c4-94d6-4077616af9c7-model-cache\") pod \"f739ec5f-a443-44c4-94d6-4077616af9c7\" (UID: \"f739ec5f-a443-44c4-94d6-4077616af9c7\") " Nov 03 00:11:44.241049 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:11:44.240983 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f739ec5f-a443-44c4-94d6-4077616af9c7-home\" (UniqueName: \"kubernetes.io/empty-dir/f739ec5f-a443-44c4-94d6-4077616af9c7-home\") pod \"f739ec5f-a443-44c4-94d6-4077616af9c7\" (UID: \"f739ec5f-a443-44c4-94d6-4077616af9c7\") " Nov 03 00:11:44.241049 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:11:44.241004 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f739ec5f-a443-44c4-94d6-4077616af9c7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f739ec5f-a443-44c4-94d6-4077616af9c7-dshm\") pod \"f739ec5f-a443-44c4-94d6-4077616af9c7\" (UID: \"f739ec5f-a443-44c4-94d6-4077616af9c7\") " Nov 03 00:11:44.241246 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:11:44.241059 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f739ec5f-a443-44c4-94d6-4077616af9c7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f739ec5f-a443-44c4-94d6-4077616af9c7-tmp-dir\") pod \"f739ec5f-a443-44c4-94d6-4077616af9c7\" (UID: \"f739ec5f-a443-44c4-94d6-4077616af9c7\") " Nov 03 00:11:44.241246 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:11:44.241091 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f739ec5f-a443-44c4-94d6-4077616af9c7-kube-api-access-jbq5l\" (UniqueName: \"kubernetes.io/projected/f739ec5f-a443-44c4-94d6-4077616af9c7-kube-api-access-jbq5l\") pod \"f739ec5f-a443-44c4-94d6-4077616af9c7\" (UID: \"f739ec5f-a443-44c4-94d6-4077616af9c7\") " Nov 03 00:11:44.241246 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:11:44.241200 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f739ec5f-a443-44c4-94d6-4077616af9c7-model-cache" pod "f739ec5f-a443-44c4-94d6-4077616af9c7" (UID: "f739ec5f-a443-44c4-94d6-4077616af9c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:11:44.241411 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:11:44.241337 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f739ec5f-a443-44c4-94d6-4077616af9c7-tmp-dir" pod "f739ec5f-a443-44c4-94d6-4077616af9c7" (UID: "f739ec5f-a443-44c4-94d6-4077616af9c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:11:44.241411 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:11:44.241360 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f739ec5f-a443-44c4-94d6-4077616af9c7-home" pod "f739ec5f-a443-44c4-94d6-4077616af9c7" (UID: "f739ec5f-a443-44c4-94d6-4077616af9c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:11:44.241542 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:11:44.241519 5727 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f739ec5f-a443-44c4-94d6-4077616af9c7-home\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:44.241587 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:11:44.241549 5727 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f739ec5f-a443-44c4-94d6-4077616af9c7-tmp-dir\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:44.241587 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:11:44.241564 5727 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f739ec5f-a443-44c4-94d6-4077616af9c7-model-cache\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:44.243510 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:11:44.243485 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f739ec5f-a443-44c4-94d6-4077616af9c7-dshm" pod "f739ec5f-a443-44c4-94d6-4077616af9c7" (UID: "f739ec5f-a443-44c4-94d6-4077616af9c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:11:44.243510 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:11:44.243496 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f739ec5f-a443-44c4-94d6-4077616af9c7-tls-certs" pod "f739ec5f-a443-44c4-94d6-4077616af9c7" (UID: "f739ec5f-a443-44c4-94d6-4077616af9c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:11:44.243666 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:11:44.243643 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f739ec5f-a443-44c4-94d6-4077616af9c7-kube-api-access-jbq5l" pod "f739ec5f-a443-44c4-94d6-4077616af9c7" (UID: "f739ec5f-a443-44c4-94d6-4077616af9c7"). InnerVolumeSpecName "kube-api-access-jbq5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:11:44.342314 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:11:44.342283 5727 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f739ec5f-a443-44c4-94d6-4077616af9c7-dshm\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:44.342314 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:11:44.342309 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbq5l\" (UniqueName: \"kubernetes.io/projected/f739ec5f-a443-44c4-94d6-4077616af9c7-kube-api-access-jbq5l\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:44.342314 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:11:44.342320 5727 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f739ec5f-a443-44c4-94d6-4077616af9c7-tls-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:45.035683 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:11:45.035650 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" event={"ID":"f739ec5f-a443-44c4-94d6-4077616af9c7","Type":"ContainerDied","Data":"54949425e63239de4a7c7be90e187b7a0ba6cbeb5d711fbb2aed7d533c186c9b"} Nov 03 00:11:45.035683 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:11:45.035680 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7" Nov 03 00:11:45.036305 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:11:45.035706 5727 scope.go:122] "RemoveContainer" containerID="2134d49b6234a845a87089d8df8314545494c466bce21e88fd3722fa5725ae27" Nov 03 00:11:45.199499 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:11:45.199451 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7"] Nov 03 00:11:45.248856 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:11:45.248830 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-3bee1b/rate-test-model-3bee1b-kserve-6cd44b7c97-fg9w7"] Nov 03 00:11:45.884394 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:11:45.884360 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f739ec5f-a443-44c4-94d6-4077616af9c7" path="/var/lib/kubelet/pods/f739ec5f-a443-44c4-94d6-4077616af9c7/volumes" Nov 03 00:11:57.845531 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:11:57.845490 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:12:27.845800 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:12:27.845763 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:12:47.043899 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:12:47.043820 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-980b82-84d578bd5f-hz792"] Nov 03 00:12:52.044832 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:12:52.044773 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-980b82-84d578bd5f-hz792" podUID="50c5d765-70d6-467e-9400-0a79f379131c" containerName="payload-pre-processing" containerID="cri-o://5fcee09b0a56aeb327f2492b2e6edf12a6acedbe56bdb282953ed4aa04c733bc" gracePeriod=25 Nov 03 00:12:54.692106 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:12:54.692071 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-980b82-84d578bd5f-hz792" podUID="50c5d765-70d6-467e-9400-0a79f379131c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:12:57.845725 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:12:57.845689 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:13:04.692001 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:04.691955 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-980b82-84d578bd5f-hz792" podUID="50c5d765-70d6-467e-9400-0a79f379131c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:13:12.294060 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:12.294035 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-980b82-84d578bd5f-hz792" Nov 03 00:13:12.384638 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:12.384607 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50c5d765-70d6-467e-9400-0a79f379131c-kube-api-access-xd7r9\" (UniqueName: \"kubernetes.io/projected/50c5d765-70d6-467e-9400-0a79f379131c-kube-api-access-xd7r9\") pod \"50c5d765-70d6-467e-9400-0a79f379131c\" (UID: \"50c5d765-70d6-467e-9400-0a79f379131c\") " Nov 03 00:13:12.384821 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:12.384675 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50c5d765-70d6-467e-9400-0a79f379131c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50c5d765-70d6-467e-9400-0a79f379131c-plugins-config-volume\") pod \"50c5d765-70d6-467e-9400-0a79f379131c\" (UID: \"50c5d765-70d6-467e-9400-0a79f379131c\") " Nov 03 00:13:12.385070 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:12.385036 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50c5d765-70d6-467e-9400-0a79f379131c-plugins-config-volume" pod "50c5d765-70d6-467e-9400-0a79f379131c" (UID: "50c5d765-70d6-467e-9400-0a79f379131c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:13:12.386885 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:12.386867 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c5d765-70d6-467e-9400-0a79f379131c-kube-api-access-xd7r9" pod "50c5d765-70d6-467e-9400-0a79f379131c" (UID: "50c5d765-70d6-467e-9400-0a79f379131c"). InnerVolumeSpecName "kube-api-access-xd7r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:13:12.485700 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:12.485662 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd7r9\" (UniqueName: \"kubernetes.io/projected/50c5d765-70d6-467e-9400-0a79f379131c-kube-api-access-xd7r9\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:13:12.485700 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:12.485691 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50c5d765-70d6-467e-9400-0a79f379131c-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:13:12.659182 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:12.659050 5727 generic.go:356] "Generic (PLEG): container finished" podID="50c5d765-70d6-467e-9400-0a79f379131c" containerID="5fcee09b0a56aeb327f2492b2e6edf12a6acedbe56bdb282953ed4aa04c733bc" exitCode=0 Nov 03 00:13:12.659182 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:12.659100 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-980b82-84d578bd5f-hz792" Nov 03 00:13:12.659182 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:12.659114 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-980b82-84d578bd5f-hz792" event={"ID":"50c5d765-70d6-467e-9400-0a79f379131c","Type":"ContainerDied","Data":"5fcee09b0a56aeb327f2492b2e6edf12a6acedbe56bdb282953ed4aa04c733bc"} Nov 03 00:13:12.659480 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:12.659189 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-980b82-84d578bd5f-hz792" event={"ID":"50c5d765-70d6-467e-9400-0a79f379131c","Type":"ContainerDied","Data":"32945a776bb31af1b15408062a530f934f8884f1d3e0f4dfdc0a26fdd5c77883"} Nov 03 00:13:12.659480 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:12.659216 5727 scope.go:122] "RemoveContainer" containerID="5fcee09b0a56aeb327f2492b2e6edf12a6acedbe56bdb282953ed4aa04c733bc" Nov 03 00:13:12.669661 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:12.669645 5727 scope.go:122] "RemoveContainer" containerID="5fcee09b0a56aeb327f2492b2e6edf12a6acedbe56bdb282953ed4aa04c733bc" Nov 03 00:13:12.669936 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:13:12.669910 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fcee09b0a56aeb327f2492b2e6edf12a6acedbe56bdb282953ed4aa04c733bc\": container with ID starting with 5fcee09b0a56aeb327f2492b2e6edf12a6acedbe56bdb282953ed4aa04c733bc not found: ID does not exist" containerID="5fcee09b0a56aeb327f2492b2e6edf12a6acedbe56bdb282953ed4aa04c733bc" Nov 03 00:13:12.669995 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:12.669945 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fcee09b0a56aeb327f2492b2e6edf12a6acedbe56bdb282953ed4aa04c733bc"} err="failed to get container status \"5fcee09b0a56aeb327f2492b2e6edf12a6acedbe56bdb282953ed4aa04c733bc\": rpc error: code = NotFound desc = could not find container \"5fcee09b0a56aeb327f2492b2e6edf12a6acedbe56bdb282953ed4aa04c733bc\": container with ID starting with 5fcee09b0a56aeb327f2492b2e6edf12a6acedbe56bdb282953ed4aa04c733bc not found: ID does not exist" Nov 03 00:13:12.829456 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:12.829412 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-980b82-84d578bd5f-hz792"] Nov 03 00:13:12.872812 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:12.872778 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-980b82-84d578bd5f-hz792"] Nov 03 00:13:13.883379 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:13.883346 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50c5d765-70d6-467e-9400-0a79f379131c" path="/var/lib/kubelet/pods/50c5d765-70d6-467e-9400-0a79f379131c/volumes" Nov 03 00:13:19.933506 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:19.933468 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv"] Nov 03 00:13:19.934427 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:19.934391 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv" podUID="6df304dc-a952-436f-b0b4-8deda4e86884" containerName="maas-api" containerID="cri-o://199ef0c98da1dab4a61d7167808cfecb7e11e70bfc6ebbb84be7de4dadc4dc48" gracePeriod=30 Nov 03 00:13:20.216063 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:20.216039 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv" Nov 03 00:13:20.261177 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:20.261152 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6df304dc-a952-436f-b0b4-8deda4e86884-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6df304dc-a952-436f-b0b4-8deda4e86884-maas-api-tls\") pod \"6df304dc-a952-436f-b0b4-8deda4e86884\" (UID: \"6df304dc-a952-436f-b0b4-8deda4e86884\") " Nov 03 00:13:20.261291 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:20.261228 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df304dc-a952-436f-b0b4-8deda4e86884-kube-api-access-j6nc2\" (UniqueName: \"kubernetes.io/projected/6df304dc-a952-436f-b0b4-8deda4e86884-kube-api-access-j6nc2\") pod \"6df304dc-a952-436f-b0b4-8deda4e86884\" (UID: \"6df304dc-a952-436f-b0b4-8deda4e86884\") " Nov 03 00:13:20.261509 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:20.261394 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6df304dc-a952-436f-b0b4-8deda4e86884-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df304dc-a952-436f-b0b4-8deda4e86884-metrics-certs\") pod \"6df304dc-a952-436f-b0b4-8deda4e86884\" (UID: \"6df304dc-a952-436f-b0b4-8deda4e86884\") " Nov 03 00:13:20.261554 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:20.261503 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6df304dc-a952-436f-b0b4-8deda4e86884-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6df304dc-a952-436f-b0b4-8deda4e86884-openshift-service-ca\") pod \"6df304dc-a952-436f-b0b4-8deda4e86884\" (UID: \"6df304dc-a952-436f-b0b4-8deda4e86884\") " Nov 03 00:13:20.261886 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:20.261857 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df304dc-a952-436f-b0b4-8deda4e86884-openshift-service-ca" pod "6df304dc-a952-436f-b0b4-8deda4e86884" (UID: "6df304dc-a952-436f-b0b4-8deda4e86884"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:13:20.261993 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:20.261967 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6df304dc-a952-436f-b0b4-8deda4e86884-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:13:20.263538 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:20.263514 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6df304dc-a952-436f-b0b4-8deda4e86884-maas-api-tls" pod "6df304dc-a952-436f-b0b4-8deda4e86884" (UID: "6df304dc-a952-436f-b0b4-8deda4e86884"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:13:20.264009 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:20.263985 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df304dc-a952-436f-b0b4-8deda4e86884-kube-api-access-j6nc2" pod "6df304dc-a952-436f-b0b4-8deda4e86884" (UID: "6df304dc-a952-436f-b0b4-8deda4e86884"). InnerVolumeSpecName "kube-api-access-j6nc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:13:20.264009 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:20.263980 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6df304dc-a952-436f-b0b4-8deda4e86884-metrics-certs" pod "6df304dc-a952-436f-b0b4-8deda4e86884" (UID: "6df304dc-a952-436f-b0b4-8deda4e86884"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:13:20.362681 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:20.362642 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6nc2\" (UniqueName: \"kubernetes.io/projected/6df304dc-a952-436f-b0b4-8deda4e86884-kube-api-access-j6nc2\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:13:20.362681 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:20.362675 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df304dc-a952-436f-b0b4-8deda4e86884-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:13:20.362681 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:20.362688 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6df304dc-a952-436f-b0b4-8deda4e86884-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:13:20.716863 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:20.716830 5727 generic.go:356] "Generic (PLEG): container finished" podID="6df304dc-a952-436f-b0b4-8deda4e86884" containerID="199ef0c98da1dab4a61d7167808cfecb7e11e70bfc6ebbb84be7de4dadc4dc48" exitCode=0 Nov 03 00:13:20.717054 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:20.716876 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv" Nov 03 00:13:20.717054 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:20.716912 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv" event={"ID":"6df304dc-a952-436f-b0b4-8deda4e86884","Type":"ContainerDied","Data":"199ef0c98da1dab4a61d7167808cfecb7e11e70bfc6ebbb84be7de4dadc4dc48"} Nov 03 00:13:20.717054 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:20.716980 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv" event={"ID":"6df304dc-a952-436f-b0b4-8deda4e86884","Type":"ContainerDied","Data":"e1f08eb386e34f1da4ceebdfe045974bb6ce7fc089918054014bc55534b30f69"} Nov 03 00:13:20.717054 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:20.717016 5727 scope.go:122] "RemoveContainer" containerID="199ef0c98da1dab4a61d7167808cfecb7e11e70bfc6ebbb84be7de4dadc4dc48" Nov 03 00:13:20.728746 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:20.728729 5727 scope.go:122] "RemoveContainer" containerID="199ef0c98da1dab4a61d7167808cfecb7e11e70bfc6ebbb84be7de4dadc4dc48" Nov 03 00:13:20.729015 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:13:20.728986 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"199ef0c98da1dab4a61d7167808cfecb7e11e70bfc6ebbb84be7de4dadc4dc48\": container with ID starting with 199ef0c98da1dab4a61d7167808cfecb7e11e70bfc6ebbb84be7de4dadc4dc48 not found: ID does not exist" containerID="199ef0c98da1dab4a61d7167808cfecb7e11e70bfc6ebbb84be7de4dadc4dc48" Nov 03 00:13:20.729086 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:20.729021 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"199ef0c98da1dab4a61d7167808cfecb7e11e70bfc6ebbb84be7de4dadc4dc48"} err="failed to get container status \"199ef0c98da1dab4a61d7167808cfecb7e11e70bfc6ebbb84be7de4dadc4dc48\": rpc error: code = NotFound desc = could not find container \"199ef0c98da1dab4a61d7167808cfecb7e11e70bfc6ebbb84be7de4dadc4dc48\": container with ID starting with 199ef0c98da1dab4a61d7167808cfecb7e11e70bfc6ebbb84be7de4dadc4dc48 not found: ID does not exist" Nov 03 00:13:20.881370 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:20.881332 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv"] Nov 03 00:13:20.885001 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:20.884973 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-980b82-6888fdbf67-qckqv"] Nov 03 00:13:21.883053 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:21.883009 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6df304dc-a952-436f-b0b4-8deda4e86884" path="/var/lib/kubelet/pods/6df304dc-a952-436f-b0b4-8deda4e86884/volumes" Nov 03 00:13:27.848478 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:13:27.848434 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:13:32.383885 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:32.383849 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8d3cb-679d85bcf-5mkkh"] Nov 03 00:13:35.020745 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:35.020710 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 03 00:13:35.037000 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:35.036978 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 03 00:13:35.131093 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:35.131061 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 03 00:13:35.146814 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:35.146788 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 03 00:13:37.385228 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:37.385170 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8d3cb-679d85bcf-5mkkh" podUID="6206423f-eac2-4922-89fe-f5632e0fe671" containerName="payload-pre-processing" containerID="cri-o://3c3615939cca7c765ae7a7ae1f7ad2aa9e1eb3fba871d49d95c709c5cf518055" gracePeriod=25 Nov 03 00:13:46.924898 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:46.924859 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8d3cb-679d85bcf-5mkkh" podUID="6206423f-eac2-4922-89fe-f5632e0fe671" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:13:56.925425 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:56.925384 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8d3cb-679d85bcf-5mkkh" podUID="6206423f-eac2-4922-89fe-f5632e0fe671" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:13:57.656369 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:57.656307 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8d3cb-679d85bcf-5mkkh" Nov 03 00:13:57.742506 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:57.742478 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6206423f-eac2-4922-89fe-f5632e0fe671-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6206423f-eac2-4922-89fe-f5632e0fe671-plugins-config-volume\") pod \"6206423f-eac2-4922-89fe-f5632e0fe671\" (UID: \"6206423f-eac2-4922-89fe-f5632e0fe671\") " Nov 03 00:13:57.742678 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:57.742530 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6206423f-eac2-4922-89fe-f5632e0fe671-kube-api-access-6fbzv\" (UniqueName: \"kubernetes.io/projected/6206423f-eac2-4922-89fe-f5632e0fe671-kube-api-access-6fbzv\") pod \"6206423f-eac2-4922-89fe-f5632e0fe671\" (UID: \"6206423f-eac2-4922-89fe-f5632e0fe671\") " Nov 03 00:13:57.742866 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:57.742840 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6206423f-eac2-4922-89fe-f5632e0fe671-plugins-config-volume" pod "6206423f-eac2-4922-89fe-f5632e0fe671" (UID: "6206423f-eac2-4922-89fe-f5632e0fe671"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:13:57.744979 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:57.744947 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6206423f-eac2-4922-89fe-f5632e0fe671-kube-api-access-6fbzv" pod "6206423f-eac2-4922-89fe-f5632e0fe671" (UID: "6206423f-eac2-4922-89fe-f5632e0fe671"). InnerVolumeSpecName "kube-api-access-6fbzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:13:57.843955 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:57.843916 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6206423f-eac2-4922-89fe-f5632e0fe671-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:13:57.843955 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:57.843953 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fbzv\" (UniqueName: \"kubernetes.io/projected/6206423f-eac2-4922-89fe-f5632e0fe671-kube-api-access-6fbzv\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:13:57.846686 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:13:57.846654 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:13:57.977065 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:57.976985 5727 generic.go:356] "Generic (PLEG): container finished" podID="6206423f-eac2-4922-89fe-f5632e0fe671" containerID="3c3615939cca7c765ae7a7ae1f7ad2aa9e1eb3fba871d49d95c709c5cf518055" exitCode=0 Nov 03 00:13:57.977065 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:57.977030 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8d3cb-679d85bcf-5mkkh" Nov 03 00:13:57.977604 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:57.977058 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8d3cb-679d85bcf-5mkkh" event={"ID":"6206423f-eac2-4922-89fe-f5632e0fe671","Type":"ContainerDied","Data":"3c3615939cca7c765ae7a7ae1f7ad2aa9e1eb3fba871d49d95c709c5cf518055"} Nov 03 00:13:57.977604 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:57.977111 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8d3cb-679d85bcf-5mkkh" event={"ID":"6206423f-eac2-4922-89fe-f5632e0fe671","Type":"ContainerDied","Data":"0cf7ed1f68797fc9e8e26912cd635042ea0ddf97d8f9ff99c925f449ee3861eb"} Nov 03 00:13:57.977604 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:57.977152 5727 scope.go:122] "RemoveContainer" containerID="3c3615939cca7c765ae7a7ae1f7ad2aa9e1eb3fba871d49d95c709c5cf518055" Nov 03 00:13:57.987517 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:57.987494 5727 scope.go:122] "RemoveContainer" containerID="3c3615939cca7c765ae7a7ae1f7ad2aa9e1eb3fba871d49d95c709c5cf518055" Nov 03 00:13:57.987802 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:13:57.987781 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c3615939cca7c765ae7a7ae1f7ad2aa9e1eb3fba871d49d95c709c5cf518055\": container with ID starting with 3c3615939cca7c765ae7a7ae1f7ad2aa9e1eb3fba871d49d95c709c5cf518055 not found: ID does not exist" containerID="3c3615939cca7c765ae7a7ae1f7ad2aa9e1eb3fba871d49d95c709c5cf518055" Nov 03 00:13:57.987871 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:57.987811 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c3615939cca7c765ae7a7ae1f7ad2aa9e1eb3fba871d49d95c709c5cf518055"} err="failed to get container status \"3c3615939cca7c765ae7a7ae1f7ad2aa9e1eb3fba871d49d95c709c5cf518055\": rpc error: code = NotFound desc = could not find container \"3c3615939cca7c765ae7a7ae1f7ad2aa9e1eb3fba871d49d95c709c5cf518055\": container with ID starting with 3c3615939cca7c765ae7a7ae1f7ad2aa9e1eb3fba871d49d95c709c5cf518055 not found: ID does not exist" Nov 03 00:13:58.141292 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:58.141256 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8d3cb-679d85bcf-5mkkh"] Nov 03 00:13:58.145009 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:58.144986 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b8d3cb-679d85bcf-5mkkh"] Nov 03 00:13:59.884647 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:13:59.884614 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6206423f-eac2-4922-89fe-f5632e0fe671" path="/var/lib/kubelet/pods/6206423f-eac2-4922-89fe-f5632e0fe671/volumes" Nov 03 00:14:09.574664 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:09.574623 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn"] Nov 03 00:14:18.920673 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:18.920574 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-08c2c6-7b8bd45695-rf87d"] Nov 03 00:14:19.027353 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:19.027319 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-08c2c6-7bcc7fc5dc-59fbq"] Nov 03 00:14:23.921406 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:23.921344 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-08c2c6-7b8bd45695-rf87d" podUID="2158b9be-b6e4-4a86-8c9b-441abe53d862" containerName="payload-processing" containerID="cri-o://a508af4eb526befd1e233e6e87cfb358a6cbb646bac02982a0141e8b976733e4" gracePeriod=25 Nov 03 00:14:24.028503 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:24.028460 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-08c2c6-7bcc7fc5dc-59fbq" podUID="73abdc63-f4c0-48b5-913b-136f46987d0e" containerName="payload-pre-processing" containerID="cri-o://7cc0ea899a8f3af38ff939dd7c7a510e441b494d15102d09c114d803e9aae1b6" gracePeriod=25 Nov 03 00:14:24.599143 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:24.599094 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" podUID="7ef6e83d-2b3b-44c2-a973-86f2587ac616" containerName="main" containerID="cri-o://de9ff54c6f30c6c8ce4ebf978b30b1e69a2db6aa3ab286efb5aca880fad54c2b" gracePeriod=45 Nov 03 00:14:24.886419 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:24.886392 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" Nov 03 00:14:25.029218 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:25.029172 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ef6e83d-2b3b-44c2-a973-86f2587ac616-tls-certs\" (UniqueName: \"kubernetes.io/secret/7ef6e83d-2b3b-44c2-a973-86f2587ac616-tls-certs\") pod \"7ef6e83d-2b3b-44c2-a973-86f2587ac616\" (UID: \"7ef6e83d-2b3b-44c2-a973-86f2587ac616\") " Nov 03 00:14:25.029705 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:25.029280 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ef6e83d-2b3b-44c2-a973-86f2587ac616-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ef6e83d-2b3b-44c2-a973-86f2587ac616-dshm\") pod \"7ef6e83d-2b3b-44c2-a973-86f2587ac616\" (UID: \"7ef6e83d-2b3b-44c2-a973-86f2587ac616\") " Nov 03 00:14:25.029705 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:25.029336 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ef6e83d-2b3b-44c2-a973-86f2587ac616-kube-api-access-kkvwp\" (UniqueName: \"kubernetes.io/projected/7ef6e83d-2b3b-44c2-a973-86f2587ac616-kube-api-access-kkvwp\") pod \"7ef6e83d-2b3b-44c2-a973-86f2587ac616\" (UID: \"7ef6e83d-2b3b-44c2-a973-86f2587ac616\") " Nov 03 00:14:25.029705 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:25.029392 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ef6e83d-2b3b-44c2-a973-86f2587ac616-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ef6e83d-2b3b-44c2-a973-86f2587ac616-model-cache\") pod \"7ef6e83d-2b3b-44c2-a973-86f2587ac616\" (UID: \"7ef6e83d-2b3b-44c2-a973-86f2587ac616\") " Nov 03 00:14:25.029705 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:25.029433 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ef6e83d-2b3b-44c2-a973-86f2587ac616-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ef6e83d-2b3b-44c2-a973-86f2587ac616-tmp-dir\") pod \"7ef6e83d-2b3b-44c2-a973-86f2587ac616\" (UID: \"7ef6e83d-2b3b-44c2-a973-86f2587ac616\") " Nov 03 00:14:25.029705 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:25.029501 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ef6e83d-2b3b-44c2-a973-86f2587ac616-home\" (UniqueName: \"kubernetes.io/empty-dir/7ef6e83d-2b3b-44c2-a973-86f2587ac616-home\") pod \"7ef6e83d-2b3b-44c2-a973-86f2587ac616\" (UID: \"7ef6e83d-2b3b-44c2-a973-86f2587ac616\") " Nov 03 00:14:25.030068 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:25.029836 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef6e83d-2b3b-44c2-a973-86f2587ac616-model-cache" pod "7ef6e83d-2b3b-44c2-a973-86f2587ac616" (UID: "7ef6e83d-2b3b-44c2-a973-86f2587ac616"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:25.030068 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:25.029983 5727 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ef6e83d-2b3b-44c2-a973-86f2587ac616-model-cache\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:25.030068 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:25.030072 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef6e83d-2b3b-44c2-a973-86f2587ac616-home" pod "7ef6e83d-2b3b-44c2-a973-86f2587ac616" (UID: "7ef6e83d-2b3b-44c2-a973-86f2587ac616"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:25.030427 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:25.030232 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef6e83d-2b3b-44c2-a973-86f2587ac616-tmp-dir" pod "7ef6e83d-2b3b-44c2-a973-86f2587ac616" (UID: "7ef6e83d-2b3b-44c2-a973-86f2587ac616"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:25.031779 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:25.031748 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef6e83d-2b3b-44c2-a973-86f2587ac616-dshm" pod "7ef6e83d-2b3b-44c2-a973-86f2587ac616" (UID: "7ef6e83d-2b3b-44c2-a973-86f2587ac616"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:25.032313 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:25.032265 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ef6e83d-2b3b-44c2-a973-86f2587ac616-kube-api-access-kkvwp" pod "7ef6e83d-2b3b-44c2-a973-86f2587ac616" (UID: "7ef6e83d-2b3b-44c2-a973-86f2587ac616"). InnerVolumeSpecName "kube-api-access-kkvwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:14:25.032415 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:25.032394 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ef6e83d-2b3b-44c2-a973-86f2587ac616-tls-certs" pod "7ef6e83d-2b3b-44c2-a973-86f2587ac616" (UID: "7ef6e83d-2b3b-44c2-a973-86f2587ac616"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:14:25.131605 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:25.131512 5727 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ef6e83d-2b3b-44c2-a973-86f2587ac616-tls-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:25.131605 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:25.131544 5727 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ef6e83d-2b3b-44c2-a973-86f2587ac616-dshm\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:25.131605 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:25.131558 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkvwp\" (UniqueName: \"kubernetes.io/projected/7ef6e83d-2b3b-44c2-a973-86f2587ac616-kube-api-access-kkvwp\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:25.131605 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:25.131571 5727 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ef6e83d-2b3b-44c2-a973-86f2587ac616-tmp-dir\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:25.131605 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:25.131584 5727 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ef6e83d-2b3b-44c2-a973-86f2587ac616-home\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:25.160186 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:25.160109 5727 generic.go:356] "Generic (PLEG): container finished" podID="7ef6e83d-2b3b-44c2-a973-86f2587ac616" containerID="de9ff54c6f30c6c8ce4ebf978b30b1e69a2db6aa3ab286efb5aca880fad54c2b" exitCode=0 Nov 03 00:14:25.160376 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:25.160213 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" Nov 03 00:14:25.160376 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:25.160214 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" event={"ID":"7ef6e83d-2b3b-44c2-a973-86f2587ac616","Type":"ContainerDied","Data":"de9ff54c6f30c6c8ce4ebf978b30b1e69a2db6aa3ab286efb5aca880fad54c2b"} Nov 03 00:14:25.160376 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:25.160326 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn" event={"ID":"7ef6e83d-2b3b-44c2-a973-86f2587ac616","Type":"ContainerDied","Data":"21fd918261c4a9807af472c1616f3e8baa2d95eca9b0fc0078a13f8bc3797877"} Nov 03 00:14:25.160376 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:25.160348 5727 scope.go:122] "RemoveContainer" containerID="de9ff54c6f30c6c8ce4ebf978b30b1e69a2db6aa3ab286efb5aca880fad54c2b" Nov 03 00:14:25.173439 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:25.173415 5727 scope.go:122] "RemoveContainer" containerID="de9ff54c6f30c6c8ce4ebf978b30b1e69a2db6aa3ab286efb5aca880fad54c2b" Nov 03 00:14:25.173795 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:14:25.173775 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de9ff54c6f30c6c8ce4ebf978b30b1e69a2db6aa3ab286efb5aca880fad54c2b\": container with ID starting with de9ff54c6f30c6c8ce4ebf978b30b1e69a2db6aa3ab286efb5aca880fad54c2b not found: ID does not exist" containerID="de9ff54c6f30c6c8ce4ebf978b30b1e69a2db6aa3ab286efb5aca880fad54c2b" Nov 03 00:14:25.173879 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:25.173810 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de9ff54c6f30c6c8ce4ebf978b30b1e69a2db6aa3ab286efb5aca880fad54c2b"} err="failed to get container status \"de9ff54c6f30c6c8ce4ebf978b30b1e69a2db6aa3ab286efb5aca880fad54c2b\": rpc error: code = NotFound desc = could not find container \"de9ff54c6f30c6c8ce4ebf978b30b1e69a2db6aa3ab286efb5aca880fad54c2b\": container with ID starting with de9ff54c6f30c6c8ce4ebf978b30b1e69a2db6aa3ab286efb5aca880fad54c2b not found: ID does not exist" Nov 03 00:14:25.324730 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:25.324702 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn"] Nov 03 00:14:25.334062 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:25.334030 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/distinct-w2-08c2c6-kserve-7dc6c6d9c5-qt9zn"] Nov 03 00:14:25.883750 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:25.883718 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ef6e83d-2b3b-44c2-a973-86f2587ac616" path="/var/lib/kubelet/pods/7ef6e83d-2b3b-44c2-a973-86f2587ac616/volumes" Nov 03 00:14:27.846599 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:14:27.846554 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:14:31.681233 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:31.681194 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-08c2c6-7bcc7fc5dc-59fbq" podUID="73abdc63-f4c0-48b5-913b-136f46987d0e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:14:31.681647 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:31.681194 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-08c2c6-7b8bd45695-rf87d" podUID="2158b9be-b6e4-4a86-8c9b-441abe53d862" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:14:37.322866 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:37.322833 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp"] Nov 03 00:14:37.323385 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:37.323031 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp" podUID="49503b5c-335b-4faa-b09e-4a5fa861033d" containerName="maas-api" containerID="cri-o://840610165cb0383613f961487561402f3cf93a3694a7911630dc8c8562afca77" gracePeriod=30 Nov 03 00:14:37.580625 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:37.580571 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp" Nov 03 00:14:37.676028 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:37.675990 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49503b5c-335b-4faa-b09e-4a5fa861033d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49503b5c-335b-4faa-b09e-4a5fa861033d-maas-api-tls\") pod \"49503b5c-335b-4faa-b09e-4a5fa861033d\" (UID: \"49503b5c-335b-4faa-b09e-4a5fa861033d\") " Nov 03 00:14:37.676253 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:37.676110 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49503b5c-335b-4faa-b09e-4a5fa861033d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49503b5c-335b-4faa-b09e-4a5fa861033d-openshift-service-ca\") pod \"49503b5c-335b-4faa-b09e-4a5fa861033d\" (UID: \"49503b5c-335b-4faa-b09e-4a5fa861033d\") " Nov 03 00:14:37.676253 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:37.676184 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49503b5c-335b-4faa-b09e-4a5fa861033d-kube-api-access-mh95h\" (UniqueName: \"kubernetes.io/projected/49503b5c-335b-4faa-b09e-4a5fa861033d-kube-api-access-mh95h\") pod \"49503b5c-335b-4faa-b09e-4a5fa861033d\" (UID: \"49503b5c-335b-4faa-b09e-4a5fa861033d\") " Nov 03 00:14:37.676253 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:37.676233 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49503b5c-335b-4faa-b09e-4a5fa861033d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/49503b5c-335b-4faa-b09e-4a5fa861033d-metrics-certs\") pod \"49503b5c-335b-4faa-b09e-4a5fa861033d\" (UID: \"49503b5c-335b-4faa-b09e-4a5fa861033d\") " Nov 03 00:14:37.676490 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:37.676460 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49503b5c-335b-4faa-b09e-4a5fa861033d-openshift-service-ca" pod "49503b5c-335b-4faa-b09e-4a5fa861033d" (UID: "49503b5c-335b-4faa-b09e-4a5fa861033d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:14:37.676908 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:37.676766 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49503b5c-335b-4faa-b09e-4a5fa861033d-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:37.678448 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:37.678417 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49503b5c-335b-4faa-b09e-4a5fa861033d-maas-api-tls" pod "49503b5c-335b-4faa-b09e-4a5fa861033d" (UID: "49503b5c-335b-4faa-b09e-4a5fa861033d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:14:37.678854 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:37.678823 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49503b5c-335b-4faa-b09e-4a5fa861033d-kube-api-access-mh95h" pod "49503b5c-335b-4faa-b09e-4a5fa861033d" (UID: "49503b5c-335b-4faa-b09e-4a5fa861033d"). InnerVolumeSpecName "kube-api-access-mh95h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:14:37.678954 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:37.678862 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49503b5c-335b-4faa-b09e-4a5fa861033d-metrics-certs" pod "49503b5c-335b-4faa-b09e-4a5fa861033d" (UID: "49503b5c-335b-4faa-b09e-4a5fa861033d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:14:37.778499 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:37.778468 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh95h\" (UniqueName: \"kubernetes.io/projected/49503b5c-335b-4faa-b09e-4a5fa861033d-kube-api-access-mh95h\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:37.778499 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:37.778497 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49503b5c-335b-4faa-b09e-4a5fa861033d-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:37.778499 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:37.778508 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49503b5c-335b-4faa-b09e-4a5fa861033d-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:38.250264 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:38.250161 5727 generic.go:356] "Generic (PLEG): container finished" podID="49503b5c-335b-4faa-b09e-4a5fa861033d" containerID="840610165cb0383613f961487561402f3cf93a3694a7911630dc8c8562afca77" exitCode=0 Nov 03 00:14:38.250264 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:38.250202 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp" Nov 03 00:14:38.250264 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:38.250247 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp" event={"ID":"49503b5c-335b-4faa-b09e-4a5fa861033d","Type":"ContainerDied","Data":"840610165cb0383613f961487561402f3cf93a3694a7911630dc8c8562afca77"} Nov 03 00:14:38.250556 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:38.250306 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp" event={"ID":"49503b5c-335b-4faa-b09e-4a5fa861033d","Type":"ContainerDied","Data":"7a81a60ba1e09fd0e5d88ab78be6eabeac92271660c520fdd73b2624d0fc08de"} Nov 03 00:14:38.250556 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:38.250321 5727 scope.go:122] "RemoveContainer" containerID="840610165cb0383613f961487561402f3cf93a3694a7911630dc8c8562afca77" Nov 03 00:14:38.262280 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:38.262260 5727 scope.go:122] "RemoveContainer" containerID="840610165cb0383613f961487561402f3cf93a3694a7911630dc8c8562afca77" Nov 03 00:14:38.262551 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:14:38.262530 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"840610165cb0383613f961487561402f3cf93a3694a7911630dc8c8562afca77\": container with ID starting with 840610165cb0383613f961487561402f3cf93a3694a7911630dc8c8562afca77 not found: ID does not exist" containerID="840610165cb0383613f961487561402f3cf93a3694a7911630dc8c8562afca77" Nov 03 00:14:38.262613 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:38.262559 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"840610165cb0383613f961487561402f3cf93a3694a7911630dc8c8562afca77"} err="failed to get container status \"840610165cb0383613f961487561402f3cf93a3694a7911630dc8c8562afca77\": rpc error: code = NotFound desc = could not find container \"840610165cb0383613f961487561402f3cf93a3694a7911630dc8c8562afca77\": container with ID starting with 840610165cb0383613f961487561402f3cf93a3694a7911630dc8c8562afca77 not found: ID does not exist" Nov 03 00:14:38.413722 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:38.413691 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp"] Nov 03 00:14:38.463260 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:38.463224 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-08c2c6-58c4fbfc59-vpbnp"] Nov 03 00:14:39.885141 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:39.885088 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49503b5c-335b-4faa-b09e-4a5fa861033d" path="/var/lib/kubelet/pods/49503b5c-335b-4faa-b09e-4a5fa861033d/volumes" Nov 03 00:14:41.680965 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:41.680920 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-08c2c6-7bcc7fc5dc-59fbq" podUID="73abdc63-f4c0-48b5-913b-136f46987d0e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:14:41.681410 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:41.681089 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-08c2c6-7b8bd45695-rf87d" podUID="2158b9be-b6e4-4a86-8c9b-441abe53d862" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:14:44.192910 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:44.192842 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-08c2c6-7b8bd45695-rf87d" Nov 03 00:14:44.240436 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:44.240401 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2158b9be-b6e4-4a86-8c9b-441abe53d862-kube-api-access-2mq8m\" (UniqueName: \"kubernetes.io/projected/2158b9be-b6e4-4a86-8c9b-441abe53d862-kube-api-access-2mq8m\") pod \"2158b9be-b6e4-4a86-8c9b-441abe53d862\" (UID: \"2158b9be-b6e4-4a86-8c9b-441abe53d862\") " Nov 03 00:14:44.240592 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:44.240493 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2158b9be-b6e4-4a86-8c9b-441abe53d862-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2158b9be-b6e4-4a86-8c9b-441abe53d862-plugins-config-volume\") pod \"2158b9be-b6e4-4a86-8c9b-441abe53d862\" (UID: \"2158b9be-b6e4-4a86-8c9b-441abe53d862\") " Nov 03 00:14:44.240954 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:44.240926 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2158b9be-b6e4-4a86-8c9b-441abe53d862-plugins-config-volume" pod "2158b9be-b6e4-4a86-8c9b-441abe53d862" (UID: "2158b9be-b6e4-4a86-8c9b-441abe53d862"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:14:44.242883 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:44.242854 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2158b9be-b6e4-4a86-8c9b-441abe53d862-kube-api-access-2mq8m" pod "2158b9be-b6e4-4a86-8c9b-441abe53d862" (UID: "2158b9be-b6e4-4a86-8c9b-441abe53d862"). InnerVolumeSpecName "kube-api-access-2mq8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:14:44.289348 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:44.289325 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-08c2c6-7bcc7fc5dc-59fbq" Nov 03 00:14:44.297587 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:44.297555 5727 generic.go:356] "Generic (PLEG): container finished" podID="2158b9be-b6e4-4a86-8c9b-441abe53d862" containerID="a508af4eb526befd1e233e6e87cfb358a6cbb646bac02982a0141e8b976733e4" exitCode=0 Nov 03 00:14:44.297700 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:44.297589 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-08c2c6-7b8bd45695-rf87d" event={"ID":"2158b9be-b6e4-4a86-8c9b-441abe53d862","Type":"ContainerDied","Data":"a508af4eb526befd1e233e6e87cfb358a6cbb646bac02982a0141e8b976733e4"} Nov 03 00:14:44.297700 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:44.297639 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-08c2c6-7b8bd45695-rf87d" Nov 03 00:14:44.297700 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:44.297651 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-08c2c6-7b8bd45695-rf87d" event={"ID":"2158b9be-b6e4-4a86-8c9b-441abe53d862","Type":"ContainerDied","Data":"86000be111db86b600923364726ae722787f681e3dd439b024c970fbf9f27064"} Nov 03 00:14:44.297700 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:44.297682 5727 scope.go:122] "RemoveContainer" containerID="a508af4eb526befd1e233e6e87cfb358a6cbb646bac02982a0141e8b976733e4" Nov 03 00:14:44.299409 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:44.299382 5727 generic.go:356] "Generic (PLEG): container finished" podID="73abdc63-f4c0-48b5-913b-136f46987d0e" containerID="7cc0ea899a8f3af38ff939dd7c7a510e441b494d15102d09c114d803e9aae1b6" exitCode=0 Nov 03 00:14:44.299545 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:44.299426 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-08c2c6-7bcc7fc5dc-59fbq" event={"ID":"73abdc63-f4c0-48b5-913b-136f46987d0e","Type":"ContainerDied","Data":"7cc0ea899a8f3af38ff939dd7c7a510e441b494d15102d09c114d803e9aae1b6"} Nov 03 00:14:44.299545 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:44.299468 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-08c2c6-7bcc7fc5dc-59fbq" event={"ID":"73abdc63-f4c0-48b5-913b-136f46987d0e","Type":"ContainerDied","Data":"505df05fbb73d87235ae4ef31e2130ead7ad99e1a1ee218875b8bf79826d8e0d"} Nov 03 00:14:44.299659 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:44.299607 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-08c2c6-7bcc7fc5dc-59fbq" Nov 03 00:14:44.308609 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:44.308198 5727 scope.go:122] "RemoveContainer" containerID="a508af4eb526befd1e233e6e87cfb358a6cbb646bac02982a0141e8b976733e4" Nov 03 00:14:44.308674 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:14:44.308636 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a508af4eb526befd1e233e6e87cfb358a6cbb646bac02982a0141e8b976733e4\": container with ID starting with a508af4eb526befd1e233e6e87cfb358a6cbb646bac02982a0141e8b976733e4 not found: ID does not exist" containerID="a508af4eb526befd1e233e6e87cfb358a6cbb646bac02982a0141e8b976733e4" Nov 03 00:14:44.308729 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:44.308668 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a508af4eb526befd1e233e6e87cfb358a6cbb646bac02982a0141e8b976733e4"} err="failed to get container status \"a508af4eb526befd1e233e6e87cfb358a6cbb646bac02982a0141e8b976733e4\": rpc error: code = NotFound desc = could not find container \"a508af4eb526befd1e233e6e87cfb358a6cbb646bac02982a0141e8b976733e4\": container with ID starting with a508af4eb526befd1e233e6e87cfb358a6cbb646bac02982a0141e8b976733e4 not found: ID does not exist" Nov 03 00:14:44.308729 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:44.308690 5727 scope.go:122] "RemoveContainer" containerID="7cc0ea899a8f3af38ff939dd7c7a510e441b494d15102d09c114d803e9aae1b6" Nov 03 00:14:44.322214 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:44.322189 5727 scope.go:122] "RemoveContainer" containerID="7cc0ea899a8f3af38ff939dd7c7a510e441b494d15102d09c114d803e9aae1b6" Nov 03 00:14:44.322523 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:14:44.322503 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cc0ea899a8f3af38ff939dd7c7a510e441b494d15102d09c114d803e9aae1b6\": container with ID starting with 7cc0ea899a8f3af38ff939dd7c7a510e441b494d15102d09c114d803e9aae1b6 not found: ID does not exist" containerID="7cc0ea899a8f3af38ff939dd7c7a510e441b494d15102d09c114d803e9aae1b6" Nov 03 00:14:44.322617 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:44.322532 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cc0ea899a8f3af38ff939dd7c7a510e441b494d15102d09c114d803e9aae1b6"} err="failed to get container status \"7cc0ea899a8f3af38ff939dd7c7a510e441b494d15102d09c114d803e9aae1b6\": rpc error: code = NotFound desc = could not find container \"7cc0ea899a8f3af38ff939dd7c7a510e441b494d15102d09c114d803e9aae1b6\": container with ID starting with 7cc0ea899a8f3af38ff939dd7c7a510e441b494d15102d09c114d803e9aae1b6 not found: ID does not exist" Nov 03 00:14:44.341815 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:44.341791 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73abdc63-f4c0-48b5-913b-136f46987d0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73abdc63-f4c0-48b5-913b-136f46987d0e-plugins-config-volume\") pod \"73abdc63-f4c0-48b5-913b-136f46987d0e\" (UID: \"73abdc63-f4c0-48b5-913b-136f46987d0e\") " Nov 03 00:14:44.341919 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:44.341841 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73abdc63-f4c0-48b5-913b-136f46987d0e-kube-api-access-j74vs\" (UniqueName: \"kubernetes.io/projected/73abdc63-f4c0-48b5-913b-136f46987d0e-kube-api-access-j74vs\") pod \"73abdc63-f4c0-48b5-913b-136f46987d0e\" (UID: \"73abdc63-f4c0-48b5-913b-136f46987d0e\") " Nov 03 00:14:44.342183 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:44.342159 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2158b9be-b6e4-4a86-8c9b-441abe53d862-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:44.342248 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:44.342198 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mq8m\" (UniqueName: \"kubernetes.io/projected/2158b9be-b6e4-4a86-8c9b-441abe53d862-kube-api-access-2mq8m\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:44.342248 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:44.342210 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73abdc63-f4c0-48b5-913b-136f46987d0e-plugins-config-volume" pod "73abdc63-f4c0-48b5-913b-136f46987d0e" (UID: "73abdc63-f4c0-48b5-913b-136f46987d0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:14:44.344103 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:44.344078 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73abdc63-f4c0-48b5-913b-136f46987d0e-kube-api-access-j74vs" pod "73abdc63-f4c0-48b5-913b-136f46987d0e" (UID: "73abdc63-f4c0-48b5-913b-136f46987d0e"). InnerVolumeSpecName "kube-api-access-j74vs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:14:44.443090 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:44.443015 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73abdc63-f4c0-48b5-913b-136f46987d0e-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:44.443090 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:44.443044 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j74vs\" (UniqueName: \"kubernetes.io/projected/73abdc63-f4c0-48b5-913b-136f46987d0e-kube-api-access-j74vs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:44.588777 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:44.588738 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-08c2c6-7b8bd45695-rf87d"] Nov 03 00:14:44.596531 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:44.596504 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-08c2c6-7b8bd45695-rf87d"] Nov 03 00:14:44.754337 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:44.754254 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-08c2c6-7bcc7fc5dc-59fbq"] Nov 03 00:14:44.764514 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:44.764478 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-08c2c6-7bcc7fc5dc-59fbq"] Nov 03 00:14:45.881868 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:45.881835 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2158b9be-b6e4-4a86-8c9b-441abe53d862" path="/var/lib/kubelet/pods/2158b9be-b6e4-4a86-8c9b-441abe53d862/volumes" Nov 03 00:14:45.882261 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:14:45.882221 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73abdc63-f4c0-48b5-913b-136f46987d0e" path="/var/lib/kubelet/pods/73abdc63-f4c0-48b5-913b-136f46987d0e/volumes" Nov 03 00:14:57.847177 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:14:57.847109 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:15:00.173231 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.173182 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894415-4rfmf"] Nov 03 00:15:00.173924 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.173901 5727 cpu_manager.go:414] "containerMap: removing container" podUID="94871baf-708f-4190-b685-87f0e8ece5d4" containerName="istio-proxy" Nov 03 00:15:00.174026 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.173925 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94871baf-708f-4190-b685-87f0e8ece5d4" containerName="istio-proxy" Nov 03 00:15:00.174026 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.173948 5727 cpu_manager.go:414] "containerMap: removing container" podUID="6df304dc-a952-436f-b0b4-8deda4e86884" containerName="maas-api" Nov 03 00:15:00.174026 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.173955 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df304dc-a952-436f-b0b4-8deda4e86884" containerName="maas-api" Nov 03 00:15:00.174026 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.173966 5727 cpu_manager.go:414] "containerMap: removing container" podUID="61ae60a1-87a0-42bf-9342-0420cd1cbce7" containerName="istio-proxy" Nov 03 00:15:00.174026 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.173973 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61ae60a1-87a0-42bf-9342-0420cd1cbce7" containerName="istio-proxy" Nov 03 00:15:00.174026 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.173991 5727 cpu_manager.go:414] "containerMap: removing container" podUID="d587d22d-31c7-4266-a7df-323f11fd4499" containerName="payload-processing" Nov 03 00:15:00.174026 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.173998 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d587d22d-31c7-4266-a7df-323f11fd4499" containerName="payload-processing" Nov 03 00:15:00.174026 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174010 5727 cpu_manager.go:414] "containerMap: removing container" podUID="6206423f-eac2-4922-89fe-f5632e0fe671" containerName="payload-pre-processing" Nov 03 00:15:00.174026 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174017 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6206423f-eac2-4922-89fe-f5632e0fe671" containerName="payload-pre-processing" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174030 5727 cpu_manager.go:414] "containerMap: removing container" podUID="926c60cb-4000-4520-bde5-f31651560cc3" containerName="payload-processing" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174037 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="926c60cb-4000-4520-bde5-f31651560cc3" containerName="payload-processing" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174047 5727 cpu_manager.go:414] "containerMap: removing container" podUID="3b693031-94ce-4930-b763-6455d23bfea8" containerName="maas-api" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174054 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b693031-94ce-4930-b763-6455d23bfea8" containerName="maas-api" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174070 5727 cpu_manager.go:414] "containerMap: removing container" podUID="7ef6e83d-2b3b-44c2-a973-86f2587ac616" containerName="main" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174077 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ef6e83d-2b3b-44c2-a973-86f2587ac616" containerName="main" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174096 5727 cpu_manager.go:414] "containerMap: removing container" podUID="49503b5c-335b-4faa-b09e-4a5fa861033d" containerName="maas-api" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174104 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49503b5c-335b-4faa-b09e-4a5fa861033d" containerName="maas-api" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174150 5727 cpu_manager.go:414] "containerMap: removing container" podUID="71f62bea-0cbe-4c6d-9cd3-63893df09a9e" containerName="istio-proxy" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174158 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71f62bea-0cbe-4c6d-9cd3-63893df09a9e" containerName="istio-proxy" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174171 5727 cpu_manager.go:414] "containerMap: removing container" podUID="5eff97c2-fd05-445f-bc27-2d5f56df0f09" containerName="maas-api" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174178 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eff97c2-fd05-445f-bc27-2d5f56df0f09" containerName="maas-api" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174191 5727 cpu_manager.go:414] "containerMap: removing container" podUID="a9048ee3-8ca0-4723-9009-f6033427b829" containerName="main" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174197 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9048ee3-8ca0-4723-9009-f6033427b829" containerName="main" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174208 5727 cpu_manager.go:414] "containerMap: removing container" podUID="50c5d765-70d6-467e-9400-0a79f379131c" containerName="payload-pre-processing" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174215 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50c5d765-70d6-467e-9400-0a79f379131c" containerName="payload-pre-processing" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174229 5727 cpu_manager.go:414] "containerMap: removing container" podUID="73abdc63-f4c0-48b5-913b-136f46987d0e" containerName="payload-pre-processing" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174236 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73abdc63-f4c0-48b5-913b-136f46987d0e" containerName="payload-pre-processing" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174247 5727 cpu_manager.go:414] "containerMap: removing container" podUID="fe18fd85-9a72-4868-8763-038ea4a0a54a" containerName="istio-proxy" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174254 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe18fd85-9a72-4868-8763-038ea4a0a54a" containerName="istio-proxy" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174276 5727 cpu_manager.go:414] "containerMap: removing container" podUID="f739ec5f-a443-44c4-94d6-4077616af9c7" containerName="main" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174282 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f739ec5f-a443-44c4-94d6-4077616af9c7" containerName="main" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174293 5727 cpu_manager.go:414] "containerMap: removing container" podUID="2158b9be-b6e4-4a86-8c9b-441abe53d862" containerName="payload-processing" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174300 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2158b9be-b6e4-4a86-8c9b-441abe53d862" containerName="payload-processing" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174433 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df304dc-a952-436f-b0b4-8deda4e86884" containerName="maas-api" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174444 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="61ae60a1-87a0-42bf-9342-0420cd1cbce7" containerName="istio-proxy" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174453 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="d587d22d-31c7-4266-a7df-323f11fd4499" containerName="payload-processing" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174465 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="6206423f-eac2-4922-89fe-f5632e0fe671" containerName="payload-pre-processing" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174476 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="926c60cb-4000-4520-bde5-f31651560cc3" containerName="payload-processing" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174485 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b693031-94ce-4930-b763-6455d23bfea8" containerName="maas-api" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174497 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ef6e83d-2b3b-44c2-a973-86f2587ac616" containerName="main" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174507 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="49503b5c-335b-4faa-b09e-4a5fa861033d" containerName="maas-api" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174520 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="71f62bea-0cbe-4c6d-9cd3-63893df09a9e" containerName="istio-proxy" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174528 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eff97c2-fd05-445f-bc27-2d5f56df0f09" containerName="maas-api" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174536 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9048ee3-8ca0-4723-9009-f6033427b829" containerName="main" Nov 03 00:15:00.174526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174545 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="50c5d765-70d6-467e-9400-0a79f379131c" containerName="payload-pre-processing" Nov 03 00:15:00.176408 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174554 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="73abdc63-f4c0-48b5-913b-136f46987d0e" containerName="payload-pre-processing" Nov 03 00:15:00.176408 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174563 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe18fd85-9a72-4868-8763-038ea4a0a54a" containerName="istio-proxy" Nov 03 00:15:00.176408 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174572 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="f739ec5f-a443-44c4-94d6-4077616af9c7" containerName="main" Nov 03 00:15:00.176408 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174580 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="2158b9be-b6e4-4a86-8c9b-441abe53d862" containerName="payload-processing" Nov 03 00:15:00.176408 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.174590 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="94871baf-708f-4190-b685-87f0e8ece5d4" containerName="istio-proxy" Nov 03 00:15:00.178467 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.178444 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn"] Nov 03 00:15:00.178611 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.178587 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-4rfmf" Nov 03 00:15:00.182156 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.182107 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn" Nov 03 00:15:00.230354 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.230327 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 03 00:15:00.284453 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.284424 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn"] Nov 03 00:15:00.288432 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.288402 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894415-4rfmf"] Nov 03 00:15:00.303902 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.303857 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmh79\" (UniqueName: \"kubernetes.io/projected/05dadd77-ff13-4a7f-9dfe-25d8290b91a6-kube-api-access-dmh79\") pod \"maas-api-key-cleanup-29894415-4rfmf\" (UID: \"05dadd77-ff13-4a7f-9dfe-25d8290b91a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-4rfmf" Nov 03 00:15:00.304041 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.303960 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tkjq\" (UniqueName: \"kubernetes.io/projected/97ac154a-3d0c-4d8c-95a8-49bdf919e5b9-kube-api-access-6tkjq\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn\" (UID: \"97ac154a-3d0c-4d8c-95a8-49bdf919e5b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn" Nov 03 00:15:00.304084 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.304068 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97ac154a-3d0c-4d8c-95a8-49bdf919e5b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn\" (UID: \"97ac154a-3d0c-4d8c-95a8-49bdf919e5b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn" Nov 03 00:15:00.304163 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.304144 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05dadd77-ff13-4a7f-9dfe-25d8290b91a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894415-4rfmf\" (UID: \"05dadd77-ff13-4a7f-9dfe-25d8290b91a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-4rfmf" Nov 03 00:15:00.404746 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.404711 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmh79\" (UniqueName: \"kubernetes.io/projected/05dadd77-ff13-4a7f-9dfe-25d8290b91a6-kube-api-access-dmh79\") pod \"maas-api-key-cleanup-29894415-4rfmf\" (UID: \"05dadd77-ff13-4a7f-9dfe-25d8290b91a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-4rfmf" Nov 03 00:15:00.404946 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.404755 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tkjq\" (UniqueName: \"kubernetes.io/projected/97ac154a-3d0c-4d8c-95a8-49bdf919e5b9-kube-api-access-6tkjq\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn\" (UID: \"97ac154a-3d0c-4d8c-95a8-49bdf919e5b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn" Nov 03 00:15:00.404946 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.404821 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97ac154a-3d0c-4d8c-95a8-49bdf919e5b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn\" (UID: \"97ac154a-3d0c-4d8c-95a8-49bdf919e5b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn" Nov 03 00:15:00.404946 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.404893 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05dadd77-ff13-4a7f-9dfe-25d8290b91a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894415-4rfmf\" (UID: \"05dadd77-ff13-4a7f-9dfe-25d8290b91a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-4rfmf" Nov 03 00:15:00.405546 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.405529 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97ac154a-3d0c-4d8c-95a8-49bdf919e5b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn\" (UID: \"97ac154a-3d0c-4d8c-95a8-49bdf919e5b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn" Nov 03 00:15:00.405744 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.405723 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05dadd77-ff13-4a7f-9dfe-25d8290b91a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894415-4rfmf\" (UID: \"05dadd77-ff13-4a7f-9dfe-25d8290b91a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-4rfmf" Nov 03 00:15:00.465429 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.465358 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tkjq\" (UniqueName: \"kubernetes.io/projected/97ac154a-3d0c-4d8c-95a8-49bdf919e5b9-kube-api-access-6tkjq\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn\" (UID: \"97ac154a-3d0c-4d8c-95a8-49bdf919e5b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn" Nov 03 00:15:00.465429 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.465403 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmh79\" (UniqueName: \"kubernetes.io/projected/05dadd77-ff13-4a7f-9dfe-25d8290b91a6-kube-api-access-dmh79\") pod \"maas-api-key-cleanup-29894415-4rfmf\" (UID: \"05dadd77-ff13-4a7f-9dfe-25d8290b91a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-4rfmf" Nov 03 00:15:00.494499 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.494472 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-4rfmf" Nov 03 00:15:00.502723 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.502705 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn" Nov 03 00:15:00.914288 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.914257 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn"] Nov 03 00:15:00.918828 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:15:00.918799 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ac154a_3d0c_4d8c_95a8_49bdf919e5b9.slice/crio-1597d840dd402db91a66a77fa5d052dc475b400ae26932a9a20778168d2fd338 WatchSource:0}: Error finding container 1597d840dd402db91a66a77fa5d052dc475b400ae26932a9a20778168d2fd338: Status 404 returned error can't find the container with id 1597d840dd402db91a66a77fa5d052dc475b400ae26932a9a20778168d2fd338 Nov 03 00:15:00.966750 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:00.966715 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894415-4rfmf"] Nov 03 00:15:01.414887 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:01.414781 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-4rfmf" event={"ID":"05dadd77-ff13-4a7f-9dfe-25d8290b91a6","Type":"ContainerStarted","Data":"8a12ff2cdce25d628f1e5e3c34e0fef42c2d78e5d6c586373836669439e38812"} Nov 03 00:15:01.414887 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:01.414856 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-4rfmf" event={"ID":"05dadd77-ff13-4a7f-9dfe-25d8290b91a6","Type":"ContainerStarted","Data":"fa6dee0c6bc10d0937da032c1e5692b99c25bf41ad69cff04b1e2fb8bd70b4ca"} Nov 03 00:15:01.416624 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:01.416586 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn" event={"ID":"97ac154a-3d0c-4d8c-95a8-49bdf919e5b9","Type":"ContainerStarted","Data":"45e3bf1bf6a4bd14778733f1c2a6021787eea5df4a14c24bc364ef14b2c44959"} Nov 03 00:15:01.416746 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:01.416640 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn" event={"ID":"97ac154a-3d0c-4d8c-95a8-49bdf919e5b9","Type":"ContainerStarted","Data":"1597d840dd402db91a66a77fa5d052dc475b400ae26932a9a20778168d2fd338"} Nov 03 00:15:01.529982 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:01.529929 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-4rfmf" podStartSLOduration=1.529916458 podStartE2EDuration="1.529916458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:01.528266862 +0000 UTC m=+5187.766764525" watchObservedRunningTime="2026-11-03 00:15:01.529916458 +0000 UTC m=+5187.768414157" Nov 03 00:15:01.643751 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:01.643694 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn" podStartSLOduration=1.6436804619999998 podStartE2EDuration="1.643680462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:01.64255558 +0000 UTC m=+5187.881053264" watchObservedRunningTime="2026-11-03 00:15:01.643680462 +0000 UTC m=+5187.882178147" Nov 03 00:15:02.425707 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:02.425674 5727 generic.go:356] "Generic (PLEG): container finished" podID="05dadd77-ff13-4a7f-9dfe-25d8290b91a6" containerID="8a12ff2cdce25d628f1e5e3c34e0fef42c2d78e5d6c586373836669439e38812" exitCode=60 Nov 03 00:15:02.426190 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:02.425756 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-4rfmf" event={"ID":"05dadd77-ff13-4a7f-9dfe-25d8290b91a6","Type":"ContainerDied","Data":"8a12ff2cdce25d628f1e5e3c34e0fef42c2d78e5d6c586373836669439e38812"} Nov 03 00:15:02.426190 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:02.425960 5727 scope.go:122] "RemoveContainer" containerID="8a12ff2cdce25d628f1e5e3c34e0fef42c2d78e5d6c586373836669439e38812" Nov 03 00:15:02.427620 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:02.427596 5727 generic.go:356] "Generic (PLEG): container finished" podID="97ac154a-3d0c-4d8c-95a8-49bdf919e5b9" containerID="45e3bf1bf6a4bd14778733f1c2a6021787eea5df4a14c24bc364ef14b2c44959" exitCode=60 Nov 03 00:15:02.427749 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:02.427706 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn" event={"ID":"97ac154a-3d0c-4d8c-95a8-49bdf919e5b9","Type":"ContainerDied","Data":"45e3bf1bf6a4bd14778733f1c2a6021787eea5df4a14c24bc364ef14b2c44959"} Nov 03 00:15:02.427893 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:02.427877 5727 scope.go:122] "RemoveContainer" containerID="45e3bf1bf6a4bd14778733f1c2a6021787eea5df4a14c24bc364ef14b2c44959" Nov 03 00:15:03.435886 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:03.435852 5727 generic.go:356] "Generic (PLEG): container finished" podID="97ac154a-3d0c-4d8c-95a8-49bdf919e5b9" containerID="7686c81e55f18aae8ce73934313622748269d35948dcc5a3854ff3c25277ac99" exitCode=60 Nov 03 00:15:03.436289 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:03.435941 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn" event={"ID":"97ac154a-3d0c-4d8c-95a8-49bdf919e5b9","Type":"ContainerDied","Data":"7686c81e55f18aae8ce73934313622748269d35948dcc5a3854ff3c25277ac99"} Nov 03 00:15:03.436289 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:03.436007 5727 scope.go:122] "RemoveContainer" containerID="45e3bf1bf6a4bd14778733f1c2a6021787eea5df4a14c24bc364ef14b2c44959" Nov 03 00:15:03.436289 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:03.436251 5727 scope.go:122] "RemoveContainer" containerID="7686c81e55f18aae8ce73934313622748269d35948dcc5a3854ff3c25277ac99" Nov 03 00:15:03.436527 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:15:03.436503 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn_odh-ai-gateway-infra(97ac154a-3d0c-4d8c-95a8-49bdf919e5b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn" podUID="97ac154a-3d0c-4d8c-95a8-49bdf919e5b9" Nov 03 00:15:03.437922 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:03.437901 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-4rfmf" event={"ID":"05dadd77-ff13-4a7f-9dfe-25d8290b91a6","Type":"ContainerStarted","Data":"3de5b16c4d9056ce915ef19a2bc3c2fe702bc4e8e042abfec213d97e9973a100"} Nov 03 00:15:04.446369 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:04.446330 5727 generic.go:356] "Generic (PLEG): container finished" podID="05dadd77-ff13-4a7f-9dfe-25d8290b91a6" containerID="3de5b16c4d9056ce915ef19a2bc3c2fe702bc4e8e042abfec213d97e9973a100" exitCode=60 Nov 03 00:15:04.446880 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:04.446423 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-4rfmf" event={"ID":"05dadd77-ff13-4a7f-9dfe-25d8290b91a6","Type":"ContainerDied","Data":"3de5b16c4d9056ce915ef19a2bc3c2fe702bc4e8e042abfec213d97e9973a100"} Nov 03 00:15:04.446880 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:04.446495 5727 scope.go:122] "RemoveContainer" containerID="8a12ff2cdce25d628f1e5e3c34e0fef42c2d78e5d6c586373836669439e38812" Nov 03 00:15:04.446880 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:04.446647 5727 scope.go:122] "RemoveContainer" containerID="3de5b16c4d9056ce915ef19a2bc3c2fe702bc4e8e042abfec213d97e9973a100" Nov 03 00:15:04.447064 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:15:04.446896 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894415-4rfmf_odh-ai-gateway-infra(05dadd77-ff13-4a7f-9dfe-25d8290b91a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-4rfmf" podUID="05dadd77-ff13-4a7f-9dfe-25d8290b91a6" Nov 03 00:15:04.448490 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:04.448474 5727 scope.go:122] "RemoveContainer" containerID="7686c81e55f18aae8ce73934313622748269d35948dcc5a3854ff3c25277ac99" Nov 03 00:15:04.448705 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:15:04.448686 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn_odh-ai-gateway-infra(97ac154a-3d0c-4d8c-95a8-49bdf919e5b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn" podUID="97ac154a-3d0c-4d8c-95a8-49bdf919e5b9" Nov 03 00:15:05.458514 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:05.458487 5727 scope.go:122] "RemoveContainer" containerID="3de5b16c4d9056ce915ef19a2bc3c2fe702bc4e8e042abfec213d97e9973a100" Nov 03 00:15:05.458916 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:15:05.458720 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894415-4rfmf_odh-ai-gateway-infra(05dadd77-ff13-4a7f-9dfe-25d8290b91a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-4rfmf" podUID="05dadd77-ff13-4a7f-9dfe-25d8290b91a6" Nov 03 00:15:13.879344 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:13.879316 5727 scope.go:122] "RemoveContainer" containerID="3de5b16c4d9056ce915ef19a2bc3c2fe702bc4e8e042abfec213d97e9973a100" Nov 03 00:15:14.520594 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:14.520512 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-4rfmf" event={"ID":"05dadd77-ff13-4a7f-9dfe-25d8290b91a6","Type":"ContainerStarted","Data":"0e352553f6447ec427e1049e6821c2c3189a722c06818c295c52932cd0704760"} Nov 03 00:15:15.529848 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:15.529808 5727 generic.go:356] "Generic (PLEG): container finished" podID="05dadd77-ff13-4a7f-9dfe-25d8290b91a6" containerID="0e352553f6447ec427e1049e6821c2c3189a722c06818c295c52932cd0704760" exitCode=60 Nov 03 00:15:15.530348 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:15.529854 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-4rfmf" event={"ID":"05dadd77-ff13-4a7f-9dfe-25d8290b91a6","Type":"ContainerDied","Data":"0e352553f6447ec427e1049e6821c2c3189a722c06818c295c52932cd0704760"} Nov 03 00:15:15.530348 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:15.529916 5727 scope.go:122] "RemoveContainer" containerID="3de5b16c4d9056ce915ef19a2bc3c2fe702bc4e8e042abfec213d97e9973a100" Nov 03 00:15:15.530348 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:15.530255 5727 scope.go:122] "RemoveContainer" containerID="0e352553f6447ec427e1049e6821c2c3189a722c06818c295c52932cd0704760" Nov 03 00:15:15.530543 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:15:15.530520 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894415-4rfmf_odh-ai-gateway-infra(05dadd77-ff13-4a7f-9dfe-25d8290b91a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-4rfmf" podUID="05dadd77-ff13-4a7f-9dfe-25d8290b91a6" Nov 03 00:15:15.644083 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:15.644057 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894415-4rfmf"] Nov 03 00:15:15.875887 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:15.875855 5727 scope.go:122] "RemoveContainer" containerID="7686c81e55f18aae8ce73934313622748269d35948dcc5a3854ff3c25277ac99" Nov 03 00:15:16.538404 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:16.538323 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn" event={"ID":"97ac154a-3d0c-4d8c-95a8-49bdf919e5b9","Type":"ContainerStarted","Data":"e7391abc4b69b1407b54cbb0a184b6264aa2c0c5379a78ee4c392ba54ade3d55"} Nov 03 00:15:16.833092 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:16.833068 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-4rfmf" Nov 03 00:15:16.875280 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:16.875246 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05dadd77-ff13-4a7f-9dfe-25d8290b91a6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05dadd77-ff13-4a7f-9dfe-25d8290b91a6-maas-api-service-ca\") pod \"05dadd77-ff13-4a7f-9dfe-25d8290b91a6\" (UID: \"05dadd77-ff13-4a7f-9dfe-25d8290b91a6\") " Nov 03 00:15:16.875430 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:16.875344 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05dadd77-ff13-4a7f-9dfe-25d8290b91a6-kube-api-access-dmh79\" (UniqueName: \"kubernetes.io/projected/05dadd77-ff13-4a7f-9dfe-25d8290b91a6-kube-api-access-dmh79\") pod \"05dadd77-ff13-4a7f-9dfe-25d8290b91a6\" (UID: \"05dadd77-ff13-4a7f-9dfe-25d8290b91a6\") " Nov 03 00:15:16.875615 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:16.875588 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05dadd77-ff13-4a7f-9dfe-25d8290b91a6-maas-api-service-ca" pod "05dadd77-ff13-4a7f-9dfe-25d8290b91a6" (UID: "05dadd77-ff13-4a7f-9dfe-25d8290b91a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:16.875785 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:16.875763 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05dadd77-ff13-4a7f-9dfe-25d8290b91a6-maas-api-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:16.877644 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:16.877621 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05dadd77-ff13-4a7f-9dfe-25d8290b91a6-kube-api-access-dmh79" pod "05dadd77-ff13-4a7f-9dfe-25d8290b91a6" (UID: "05dadd77-ff13-4a7f-9dfe-25d8290b91a6"). InnerVolumeSpecName "kube-api-access-dmh79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:16.976707 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:16.976661 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmh79\" (UniqueName: \"kubernetes.io/projected/05dadd77-ff13-4a7f-9dfe-25d8290b91a6-kube-api-access-dmh79\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:17.547794 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:17.547766 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-4rfmf" Nov 03 00:15:17.548317 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:17.547788 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-4rfmf" event={"ID":"05dadd77-ff13-4a7f-9dfe-25d8290b91a6","Type":"ContainerDied","Data":"fa6dee0c6bc10d0937da032c1e5692b99c25bf41ad69cff04b1e2fb8bd70b4ca"} Nov 03 00:15:17.548317 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:17.547854 5727 scope.go:122] "RemoveContainer" containerID="0e352553f6447ec427e1049e6821c2c3189a722c06818c295c52932cd0704760" Nov 03 00:15:17.550320 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:17.550292 5727 generic.go:356] "Generic (PLEG): container finished" podID="97ac154a-3d0c-4d8c-95a8-49bdf919e5b9" containerID="e7391abc4b69b1407b54cbb0a184b6264aa2c0c5379a78ee4c392ba54ade3d55" exitCode=60 Nov 03 00:15:17.550447 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:17.550325 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn" event={"ID":"97ac154a-3d0c-4d8c-95a8-49bdf919e5b9","Type":"ContainerDied","Data":"e7391abc4b69b1407b54cbb0a184b6264aa2c0c5379a78ee4c392ba54ade3d55"} Nov 03 00:15:17.550732 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:17.550706 5727 scope.go:122] "RemoveContainer" containerID="e7391abc4b69b1407b54cbb0a184b6264aa2c0c5379a78ee4c392ba54ade3d55" Nov 03 00:15:17.551012 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:15:17.550986 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn_odh-ai-gateway-infra(97ac154a-3d0c-4d8c-95a8-49bdf919e5b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn" podUID="97ac154a-3d0c-4d8c-95a8-49bdf919e5b9" Nov 03 00:15:17.570903 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:17.570871 5727 scope.go:122] "RemoveContainer" containerID="7686c81e55f18aae8ce73934313622748269d35948dcc5a3854ff3c25277ac99" Nov 03 00:15:17.663294 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:17.663265 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn"] Nov 03 00:15:17.826989 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:17.826953 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894415-4rfmf"] Nov 03 00:15:17.830939 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:17.830909 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894415-4rfmf"] Nov 03 00:15:17.884055 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:17.884016 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05dadd77-ff13-4a7f-9dfe-25d8290b91a6" path="/var/lib/kubelet/pods/05dadd77-ff13-4a7f-9dfe-25d8290b91a6/volumes" Nov 03 00:15:18.715474 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:18.715452 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn" Nov 03 00:15:18.795792 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:18.795756 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97ac154a-3d0c-4d8c-95a8-49bdf919e5b9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97ac154a-3d0c-4d8c-95a8-49bdf919e5b9-maas-api-service-ca\") pod \"97ac154a-3d0c-4d8c-95a8-49bdf919e5b9\" (UID: \"97ac154a-3d0c-4d8c-95a8-49bdf919e5b9\") " Nov 03 00:15:18.795957 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:18.795816 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97ac154a-3d0c-4d8c-95a8-49bdf919e5b9-kube-api-access-6tkjq\" (UniqueName: \"kubernetes.io/projected/97ac154a-3d0c-4d8c-95a8-49bdf919e5b9-kube-api-access-6tkjq\") pod \"97ac154a-3d0c-4d8c-95a8-49bdf919e5b9\" (UID: \"97ac154a-3d0c-4d8c-95a8-49bdf919e5b9\") " Nov 03 00:15:18.796217 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:18.796191 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97ac154a-3d0c-4d8c-95a8-49bdf919e5b9-maas-api-service-ca" pod "97ac154a-3d0c-4d8c-95a8-49bdf919e5b9" (UID: "97ac154a-3d0c-4d8c-95a8-49bdf919e5b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:18.798168 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:18.798116 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ac154a-3d0c-4d8c-95a8-49bdf919e5b9-kube-api-access-6tkjq" pod "97ac154a-3d0c-4d8c-95a8-49bdf919e5b9" (UID: "97ac154a-3d0c-4d8c-95a8-49bdf919e5b9"). InnerVolumeSpecName "kube-api-access-6tkjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:18.897318 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:18.897283 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97ac154a-3d0c-4d8c-95a8-49bdf919e5b9-maas-api-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:18.897318 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:18.897314 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tkjq\" (UniqueName: \"kubernetes.io/projected/97ac154a-3d0c-4d8c-95a8-49bdf919e5b9-kube-api-access-6tkjq\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:19.569413 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:19.569379 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn" event={"ID":"97ac154a-3d0c-4d8c-95a8-49bdf919e5b9","Type":"ContainerDied","Data":"1597d840dd402db91a66a77fa5d052dc475b400ae26932a9a20778168d2fd338"} Nov 03 00:15:19.569608 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:19.569418 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn" Nov 03 00:15:19.569608 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:19.569444 5727 scope.go:122] "RemoveContainer" containerID="e7391abc4b69b1407b54cbb0a184b6264aa2c0c5379a78ee4c392ba54ade3d55" Nov 03 00:15:19.793745 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:19.793712 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn"] Nov 03 00:15:19.796759 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:19.796734 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894415-hr9zn"] Nov 03 00:15:19.898636 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:19.898555 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97ac154a-3d0c-4d8c-95a8-49bdf919e5b9" path="/var/lib/kubelet/pods/97ac154a-3d0c-4d8c-95a8-49bdf919e5b9/volumes" Nov 03 00:15:26.013477 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:26.013440 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8c79816-5kg5f"] Nov 03 00:15:26.013957 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:26.013943 5727 cpu_manager.go:414] "containerMap: removing container" podUID="05dadd77-ff13-4a7f-9dfe-25d8290b91a6" containerName="cleanup" Nov 03 00:15:26.014002 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:26.013958 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05dadd77-ff13-4a7f-9dfe-25d8290b91a6" containerName="cleanup" Nov 03 00:15:26.014002 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:26.013982 5727 cpu_manager.go:414] "containerMap: removing container" podUID="97ac154a-3d0c-4d8c-95a8-49bdf919e5b9" containerName="cleanup" Nov 03 00:15:26.014002 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:26.013990 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97ac154a-3d0c-4d8c-95a8-49bdf919e5b9" containerName="cleanup" Nov 03 00:15:26.014002 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:26.014001 5727 cpu_manager.go:414] "containerMap: removing container" podUID="05dadd77-ff13-4a7f-9dfe-25d8290b91a6" containerName="cleanup" Nov 03 00:15:26.014123 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:26.014006 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05dadd77-ff13-4a7f-9dfe-25d8290b91a6" containerName="cleanup" Nov 03 00:15:26.014123 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:26.014027 5727 cpu_manager.go:414] "containerMap: removing container" podUID="97ac154a-3d0c-4d8c-95a8-49bdf919e5b9" containerName="cleanup" Nov 03 00:15:26.014123 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:26.014032 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97ac154a-3d0c-4d8c-95a8-49bdf919e5b9" containerName="cleanup" Nov 03 00:15:26.014123 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:26.014121 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="97ac154a-3d0c-4d8c-95a8-49bdf919e5b9" containerName="cleanup" Nov 03 00:15:26.014269 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:26.014145 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="97ac154a-3d0c-4d8c-95a8-49bdf919e5b9" containerName="cleanup" Nov 03 00:15:26.014269 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:26.014157 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="05dadd77-ff13-4a7f-9dfe-25d8290b91a6" containerName="cleanup" Nov 03 00:15:26.014269 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:26.014169 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="05dadd77-ff13-4a7f-9dfe-25d8290b91a6" containerName="cleanup" Nov 03 00:15:26.014269 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:26.014174 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="05dadd77-ff13-4a7f-9dfe-25d8290b91a6" containerName="cleanup" Nov 03 00:15:26.018773 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:26.018756 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c79816-5kg5f" Nov 03 00:15:26.074466 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:26.074439 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8c79816-5kg5f"] Nov 03 00:15:26.079720 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:26.079693 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/990be195-b421-4192-8136-d216d3b22e06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8c79816-5kg5f\" (UID: \"990be195-b421-4192-8136-d216d3b22e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c79816-5kg5f" Nov 03 00:15:26.079820 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:26.079745 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9glp\" (UniqueName: \"kubernetes.io/projected/990be195-b421-4192-8136-d216d3b22e06-kube-api-access-x9glp\") pod \"maas-api-revoke-sub-b8c79816-5kg5f\" (UID: \"990be195-b421-4192-8136-d216d3b22e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c79816-5kg5f" Nov 03 00:15:26.079820 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:26.079807 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/990be195-b421-4192-8136-d216d3b22e06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8c79816-5kg5f\" (UID: \"990be195-b421-4192-8136-d216d3b22e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c79816-5kg5f" Nov 03 00:15:26.120697 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:26.120671 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 03 00:15:26.120831 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:26.120709 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:15:26.180763 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:26.180733 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/990be195-b421-4192-8136-d216d3b22e06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8c79816-5kg5f\" (UID: \"990be195-b421-4192-8136-d216d3b22e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c79816-5kg5f" Nov 03 00:15:26.180905 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:26.180798 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/990be195-b421-4192-8136-d216d3b22e06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8c79816-5kg5f\" (UID: \"990be195-b421-4192-8136-d216d3b22e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c79816-5kg5f" Nov 03 00:15:26.180905 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:26.180845 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9glp\" (UniqueName: \"kubernetes.io/projected/990be195-b421-4192-8136-d216d3b22e06-kube-api-access-x9glp\") pod \"maas-api-revoke-sub-b8c79816-5kg5f\" (UID: \"990be195-b421-4192-8136-d216d3b22e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c79816-5kg5f" Nov 03 00:15:26.181576 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:26.181516 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/990be195-b421-4192-8136-d216d3b22e06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8c79816-5kg5f\" (UID: \"990be195-b421-4192-8136-d216d3b22e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c79816-5kg5f" Nov 03 00:15:26.183331 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:26.183310 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/990be195-b421-4192-8136-d216d3b22e06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8c79816-5kg5f\" (UID: \"990be195-b421-4192-8136-d216d3b22e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c79816-5kg5f" Nov 03 00:15:26.242076 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:26.242052 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9glp\" (UniqueName: \"kubernetes.io/projected/990be195-b421-4192-8136-d216d3b22e06-kube-api-access-x9glp\") pod \"maas-api-revoke-sub-b8c79816-5kg5f\" (UID: \"990be195-b421-4192-8136-d216d3b22e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c79816-5kg5f" Nov 03 00:15:26.331888 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:26.331861 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c79816-5kg5f" Nov 03 00:15:26.707646 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:26.707620 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8c79816-5kg5f"] Nov 03 00:15:26.709345 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:15:26.709311 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod990be195_b421_4192_8136_d216d3b22e06.slice/crio-97e6b88fb48079925d3423cd998d0532d39041e8653816a333849109926771f8 WatchSource:0}: Error finding container 97e6b88fb48079925d3423cd998d0532d39041e8653816a333849109926771f8: Status 404 returned error can't find the container with id 97e6b88fb48079925d3423cd998d0532d39041e8653816a333849109926771f8 Nov 03 00:15:27.624423 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:27.624389 5727 generic.go:356] "Generic (PLEG): container finished" podID="990be195-b421-4192-8136-d216d3b22e06" containerID="6f0a16ef00fdafe2a974c2645315cea97d8e756211614b4ba2f80157444fdb08" exitCode=0 Nov 03 00:15:27.624763 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:27.624475 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c79816-5kg5f" event={"ID":"990be195-b421-4192-8136-d216d3b22e06","Type":"ContainerDied","Data":"6f0a16ef00fdafe2a974c2645315cea97d8e756211614b4ba2f80157444fdb08"} Nov 03 00:15:27.624763 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:27.624528 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c79816-5kg5f" event={"ID":"990be195-b421-4192-8136-d216d3b22e06","Type":"ContainerStarted","Data":"97e6b88fb48079925d3423cd998d0532d39041e8653816a333849109926771f8"} Nov 03 00:15:27.845691 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:15:27.845656 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:15:28.776540 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:28.776515 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c79816-5kg5f" Nov 03 00:15:28.908528 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:28.908435 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/990be195-b421-4192-8136-d216d3b22e06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/990be195-b421-4192-8136-d216d3b22e06-maas-api-openshift-ca\") pod \"990be195-b421-4192-8136-d216d3b22e06\" (UID: \"990be195-b421-4192-8136-d216d3b22e06\") " Nov 03 00:15:28.908528 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:28.908472 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/990be195-b421-4192-8136-d216d3b22e06-kube-api-access-x9glp\" (UniqueName: \"kubernetes.io/projected/990be195-b421-4192-8136-d216d3b22e06-kube-api-access-x9glp\") pod \"990be195-b421-4192-8136-d216d3b22e06\" (UID: \"990be195-b421-4192-8136-d216d3b22e06\") " Nov 03 00:15:28.908528 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:28.908524 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/990be195-b421-4192-8136-d216d3b22e06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/990be195-b421-4192-8136-d216d3b22e06-maas-api-xks-ca\") pod \"990be195-b421-4192-8136-d216d3b22e06\" (UID: \"990be195-b421-4192-8136-d216d3b22e06\") " Nov 03 00:15:28.908841 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:28.908803 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/990be195-b421-4192-8136-d216d3b22e06-maas-api-openshift-ca" pod "990be195-b421-4192-8136-d216d3b22e06" (UID: "990be195-b421-4192-8136-d216d3b22e06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:28.910787 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:28.910759 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/990be195-b421-4192-8136-d216d3b22e06-maas-api-xks-ca" pod "990be195-b421-4192-8136-d216d3b22e06" (UID: "990be195-b421-4192-8136-d216d3b22e06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:15:28.910933 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:28.910897 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/990be195-b421-4192-8136-d216d3b22e06-kube-api-access-x9glp" pod "990be195-b421-4192-8136-d216d3b22e06" (UID: "990be195-b421-4192-8136-d216d3b22e06"). InnerVolumeSpecName "kube-api-access-x9glp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:29.010233 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:29.010198 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/990be195-b421-4192-8136-d216d3b22e06-maas-api-openshift-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:29.010358 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:29.010239 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9glp\" (UniqueName: \"kubernetes.io/projected/990be195-b421-4192-8136-d216d3b22e06-kube-api-access-x9glp\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:29.010358 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:29.010255 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/990be195-b421-4192-8136-d216d3b22e06-maas-api-xks-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:29.638228 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:29.638197 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c79816-5kg5f" Nov 03 00:15:29.638420 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:29.638204 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c79816-5kg5f" event={"ID":"990be195-b421-4192-8136-d216d3b22e06","Type":"ContainerDied","Data":"97e6b88fb48079925d3423cd998d0532d39041e8653816a333849109926771f8"} Nov 03 00:15:29.638420 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:29.638333 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97e6b88fb48079925d3423cd998d0532d39041e8653816a333849109926771f8" Nov 03 00:15:37.624777 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:37.624743 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ee97e86-tqnrv"] Nov 03 00:15:37.625238 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:37.625224 5727 cpu_manager.go:414] "containerMap: removing container" podUID="05dadd77-ff13-4a7f-9dfe-25d8290b91a6" containerName="cleanup" Nov 03 00:15:37.625283 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:37.625239 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05dadd77-ff13-4a7f-9dfe-25d8290b91a6" containerName="cleanup" Nov 03 00:15:37.625283 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:37.625276 5727 cpu_manager.go:414] "containerMap: removing container" podUID="990be195-b421-4192-8136-d216d3b22e06" containerName="revoke-keys" Nov 03 00:15:37.625283 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:37.625281 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="990be195-b421-4192-8136-d216d3b22e06" containerName="revoke-keys" Nov 03 00:15:37.625389 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:37.625294 5727 cpu_manager.go:414] "containerMap: removing container" podUID="97ac154a-3d0c-4d8c-95a8-49bdf919e5b9" containerName="cleanup" Nov 03 00:15:37.625389 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:37.625299 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97ac154a-3d0c-4d8c-95a8-49bdf919e5b9" containerName="cleanup" Nov 03 00:15:37.625389 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:37.625371 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="990be195-b421-4192-8136-d216d3b22e06" containerName="revoke-keys" Nov 03 00:15:37.625389 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:37.625379 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="97ac154a-3d0c-4d8c-95a8-49bdf919e5b9" containerName="cleanup" Nov 03 00:15:37.629995 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:37.629976 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee97e86-tqnrv" Nov 03 00:15:37.682756 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:37.682724 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:15:37.732026 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:37.731995 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 03 00:15:37.736514 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:37.736488 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ee97e86-tqnrv"] Nov 03 00:15:37.797552 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:37.797521 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4wwr\" (UniqueName: \"kubernetes.io/projected/4dd412c7-c93e-4cbc-bf1d-2708537a8c25-kube-api-access-n4wwr\") pod \"maas-api-revoke-sub-2ee97e86-tqnrv\" (UID: \"4dd412c7-c93e-4cbc-bf1d-2708537a8c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee97e86-tqnrv" Nov 03 00:15:37.797705 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:37.797658 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dd412c7-c93e-4cbc-bf1d-2708537a8c25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ee97e86-tqnrv\" (UID: \"4dd412c7-c93e-4cbc-bf1d-2708537a8c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee97e86-tqnrv" Nov 03 00:15:37.797819 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:37.797788 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dd412c7-c93e-4cbc-bf1d-2708537a8c25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ee97e86-tqnrv\" (UID: \"4dd412c7-c93e-4cbc-bf1d-2708537a8c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee97e86-tqnrv" Nov 03 00:15:37.898753 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:37.898680 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4wwr\" (UniqueName: \"kubernetes.io/projected/4dd412c7-c93e-4cbc-bf1d-2708537a8c25-kube-api-access-n4wwr\") pod \"maas-api-revoke-sub-2ee97e86-tqnrv\" (UID: \"4dd412c7-c93e-4cbc-bf1d-2708537a8c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee97e86-tqnrv" Nov 03 00:15:37.898753 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:37.898740 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dd412c7-c93e-4cbc-bf1d-2708537a8c25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ee97e86-tqnrv\" (UID: \"4dd412c7-c93e-4cbc-bf1d-2708537a8c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee97e86-tqnrv" Nov 03 00:15:37.898936 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:37.898790 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dd412c7-c93e-4cbc-bf1d-2708537a8c25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ee97e86-tqnrv\" (UID: \"4dd412c7-c93e-4cbc-bf1d-2708537a8c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee97e86-tqnrv" Nov 03 00:15:37.899539 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:37.899521 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dd412c7-c93e-4cbc-bf1d-2708537a8c25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ee97e86-tqnrv\" (UID: \"4dd412c7-c93e-4cbc-bf1d-2708537a8c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee97e86-tqnrv" Nov 03 00:15:37.901331 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:37.901316 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dd412c7-c93e-4cbc-bf1d-2708537a8c25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ee97e86-tqnrv\" (UID: \"4dd412c7-c93e-4cbc-bf1d-2708537a8c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee97e86-tqnrv" Nov 03 00:15:37.959541 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:37.959509 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4wwr\" (UniqueName: \"kubernetes.io/projected/4dd412c7-c93e-4cbc-bf1d-2708537a8c25-kube-api-access-n4wwr\") pod \"maas-api-revoke-sub-2ee97e86-tqnrv\" (UID: \"4dd412c7-c93e-4cbc-bf1d-2708537a8c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee97e86-tqnrv" Nov 03 00:15:38.243622 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:38.243543 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee97e86-tqnrv" Nov 03 00:15:38.619702 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:38.619678 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ee97e86-tqnrv"] Nov 03 00:15:38.622041 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:15:38.622017 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dd412c7_c93e_4cbc_bf1d_2708537a8c25.slice/crio-ba424fcb9394eef251259cdf3bde090baea0c0a0f11989191445c8363c09d9a4 WatchSource:0}: Error finding container ba424fcb9394eef251259cdf3bde090baea0c0a0f11989191445c8363c09d9a4: Status 404 returned error can't find the container with id ba424fcb9394eef251259cdf3bde090baea0c0a0f11989191445c8363c09d9a4 Nov 03 00:15:38.702532 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:38.702492 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee97e86-tqnrv" event={"ID":"4dd412c7-c93e-4cbc-bf1d-2708537a8c25","Type":"ContainerStarted","Data":"ba424fcb9394eef251259cdf3bde090baea0c0a0f11989191445c8363c09d9a4"} Nov 03 00:15:39.710834 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:39.710802 5727 generic.go:356] "Generic (PLEG): container finished" podID="4dd412c7-c93e-4cbc-bf1d-2708537a8c25" containerID="fbc7858dbe5fccf479eada8e3f7c0d4e7b8d7f5e43c8e5794eb363ea0bd64800" exitCode=0 Nov 03 00:15:39.711263 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:39.710900 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee97e86-tqnrv" event={"ID":"4dd412c7-c93e-4cbc-bf1d-2708537a8c25","Type":"ContainerDied","Data":"fbc7858dbe5fccf479eada8e3f7c0d4e7b8d7f5e43c8e5794eb363ea0bd64800"} Nov 03 00:15:40.861327 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:40.861300 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee97e86-tqnrv" Nov 03 00:15:41.035718 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:41.035632 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dd412c7-c93e-4cbc-bf1d-2708537a8c25-kube-api-access-n4wwr\" (UniqueName: \"kubernetes.io/projected/4dd412c7-c93e-4cbc-bf1d-2708537a8c25-kube-api-access-n4wwr\") pod \"4dd412c7-c93e-4cbc-bf1d-2708537a8c25\" (UID: \"4dd412c7-c93e-4cbc-bf1d-2708537a8c25\") " Nov 03 00:15:41.035862 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:41.035760 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dd412c7-c93e-4cbc-bf1d-2708537a8c25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dd412c7-c93e-4cbc-bf1d-2708537a8c25-maas-api-openshift-ca\") pod \"4dd412c7-c93e-4cbc-bf1d-2708537a8c25\" (UID: \"4dd412c7-c93e-4cbc-bf1d-2708537a8c25\") " Nov 03 00:15:41.035862 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:41.035785 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dd412c7-c93e-4cbc-bf1d-2708537a8c25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dd412c7-c93e-4cbc-bf1d-2708537a8c25-maas-api-xks-ca\") pod \"4dd412c7-c93e-4cbc-bf1d-2708537a8c25\" (UID: \"4dd412c7-c93e-4cbc-bf1d-2708537a8c25\") " Nov 03 00:15:41.036111 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:41.036089 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dd412c7-c93e-4cbc-bf1d-2708537a8c25-maas-api-openshift-ca" pod "4dd412c7-c93e-4cbc-bf1d-2708537a8c25" (UID: "4dd412c7-c93e-4cbc-bf1d-2708537a8c25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:41.037970 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:41.037944 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dd412c7-c93e-4cbc-bf1d-2708537a8c25-maas-api-xks-ca" pod "4dd412c7-c93e-4cbc-bf1d-2708537a8c25" (UID: "4dd412c7-c93e-4cbc-bf1d-2708537a8c25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:15:41.038080 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:41.038002 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd412c7-c93e-4cbc-bf1d-2708537a8c25-kube-api-access-n4wwr" pod "4dd412c7-c93e-4cbc-bf1d-2708537a8c25" (UID: "4dd412c7-c93e-4cbc-bf1d-2708537a8c25"). InnerVolumeSpecName "kube-api-access-n4wwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:41.136893 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:41.136856 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4wwr\" (UniqueName: \"kubernetes.io/projected/4dd412c7-c93e-4cbc-bf1d-2708537a8c25-kube-api-access-n4wwr\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:41.136893 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:41.136890 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dd412c7-c93e-4cbc-bf1d-2708537a8c25-maas-api-openshift-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:41.136893 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:41.136901 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dd412c7-c93e-4cbc-bf1d-2708537a8c25-maas-api-xks-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:41.726285 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:41.726249 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee97e86-tqnrv" Nov 03 00:15:41.726285 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:41.726272 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee97e86-tqnrv" event={"ID":"4dd412c7-c93e-4cbc-bf1d-2708537a8c25","Type":"ContainerDied","Data":"ba424fcb9394eef251259cdf3bde090baea0c0a0f11989191445c8363c09d9a4"} Nov 03 00:15:41.726492 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:41.726321 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba424fcb9394eef251259cdf3bde090baea0c0a0f11989191445c8363c09d9a4" Nov 03 00:15:48.496770 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:48.496676 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d31257c-2d46d"] Nov 03 00:15:48.497295 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:48.497181 5727 cpu_manager.go:414] "containerMap: removing container" podUID="4dd412c7-c93e-4cbc-bf1d-2708537a8c25" containerName="revoke-keys" Nov 03 00:15:48.497295 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:48.497208 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dd412c7-c93e-4cbc-bf1d-2708537a8c25" containerName="revoke-keys" Nov 03 00:15:48.497407 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:48.497311 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dd412c7-c93e-4cbc-bf1d-2708537a8c25" containerName="revoke-keys" Nov 03 00:15:48.500618 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:48.500596 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d31257c-2d46d" Nov 03 00:15:48.513968 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:48.513941 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrrdz\" (UniqueName: \"kubernetes.io/projected/4d26cca8-ceef-40dd-a6ce-d60d2dd34c25-kube-api-access-wrrdz\") pod \"maas-api-revoke-sub-0d31257c-2d46d\" (UID: \"4d26cca8-ceef-40dd-a6ce-d60d2dd34c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d31257c-2d46d" Nov 03 00:15:48.514069 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:48.513986 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d26cca8-ceef-40dd-a6ce-d60d2dd34c25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d31257c-2d46d\" (UID: \"4d26cca8-ceef-40dd-a6ce-d60d2dd34c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d31257c-2d46d" Nov 03 00:15:48.514112 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:48.514089 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d26cca8-ceef-40dd-a6ce-d60d2dd34c25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d31257c-2d46d\" (UID: \"4d26cca8-ceef-40dd-a6ce-d60d2dd34c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d31257c-2d46d" Nov 03 00:15:48.552692 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:48.552662 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:15:48.601829 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:48.601795 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 03 00:15:48.605506 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:48.605481 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d31257c-2d46d"] Nov 03 00:15:48.614657 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:48.614631 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrrdz\" (UniqueName: \"kubernetes.io/projected/4d26cca8-ceef-40dd-a6ce-d60d2dd34c25-kube-api-access-wrrdz\") pod \"maas-api-revoke-sub-0d31257c-2d46d\" (UID: \"4d26cca8-ceef-40dd-a6ce-d60d2dd34c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d31257c-2d46d" Nov 03 00:15:48.614789 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:48.614681 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d26cca8-ceef-40dd-a6ce-d60d2dd34c25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d31257c-2d46d\" (UID: \"4d26cca8-ceef-40dd-a6ce-d60d2dd34c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d31257c-2d46d" Nov 03 00:15:48.614789 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:48.614731 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d26cca8-ceef-40dd-a6ce-d60d2dd34c25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d31257c-2d46d\" (UID: \"4d26cca8-ceef-40dd-a6ce-d60d2dd34c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d31257c-2d46d" Nov 03 00:15:48.615522 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:48.615471 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d26cca8-ceef-40dd-a6ce-d60d2dd34c25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d31257c-2d46d\" (UID: \"4d26cca8-ceef-40dd-a6ce-d60d2dd34c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d31257c-2d46d" Nov 03 00:15:48.617382 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:48.617361 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d26cca8-ceef-40dd-a6ce-d60d2dd34c25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d31257c-2d46d\" (UID: \"4d26cca8-ceef-40dd-a6ce-d60d2dd34c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d31257c-2d46d" Nov 03 00:15:48.675377 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:48.675349 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrrdz\" (UniqueName: \"kubernetes.io/projected/4d26cca8-ceef-40dd-a6ce-d60d2dd34c25-kube-api-access-wrrdz\") pod \"maas-api-revoke-sub-0d31257c-2d46d\" (UID: \"4d26cca8-ceef-40dd-a6ce-d60d2dd34c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d31257c-2d46d" Nov 03 00:15:48.814462 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:48.814430 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d31257c-2d46d" Nov 03 00:15:49.190352 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:49.190326 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d31257c-2d46d"] Nov 03 00:15:49.192161 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:15:49.192100 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d26cca8_ceef_40dd_a6ce_d60d2dd34c25.slice/crio-c92a210f50da157d2e20b78d1495f833b03d0fb7f980634017ad0cfad1565df2 WatchSource:0}: Error finding container c92a210f50da157d2e20b78d1495f833b03d0fb7f980634017ad0cfad1565df2: Status 404 returned error can't find the container with id c92a210f50da157d2e20b78d1495f833b03d0fb7f980634017ad0cfad1565df2 Nov 03 00:15:49.780766 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:49.780680 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d31257c-2d46d" event={"ID":"4d26cca8-ceef-40dd-a6ce-d60d2dd34c25","Type":"ContainerStarted","Data":"26874028487ea372c2be868bed79a95dc2cad67a28ab66e89ee29bb8e08ba1e7"} Nov 03 00:15:49.780766 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:49.780748 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d31257c-2d46d" event={"ID":"4d26cca8-ceef-40dd-a6ce-d60d2dd34c25","Type":"ContainerStarted","Data":"c92a210f50da157d2e20b78d1495f833b03d0fb7f980634017ad0cfad1565df2"} Nov 03 00:15:49.896262 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:49.896208 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d31257c-2d46d" podStartSLOduration=1.896194324 podStartE2EDuration="1.896194324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:49.894598602 +0000 UTC m=+5236.133096290" watchObservedRunningTime="2026-11-03 00:15:49.896194324 +0000 UTC m=+5236.134691986" Nov 03 00:15:50.788324 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:50.788291 5727 generic.go:356] "Generic (PLEG): container finished" podID="4d26cca8-ceef-40dd-a6ce-d60d2dd34c25" containerID="26874028487ea372c2be868bed79a95dc2cad67a28ab66e89ee29bb8e08ba1e7" exitCode=0 Nov 03 00:15:50.788791 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:50.788381 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d31257c-2d46d" event={"ID":"4d26cca8-ceef-40dd-a6ce-d60d2dd34c25","Type":"ContainerDied","Data":"26874028487ea372c2be868bed79a95dc2cad67a28ab66e89ee29bb8e08ba1e7"} Nov 03 00:15:51.950262 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:51.950236 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d31257c-2d46d" Nov 03 00:15:52.047908 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:52.047822 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d26cca8-ceef-40dd-a6ce-d60d2dd34c25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d26cca8-ceef-40dd-a6ce-d60d2dd34c25-maas-api-openshift-ca\") pod \"4d26cca8-ceef-40dd-a6ce-d60d2dd34c25\" (UID: \"4d26cca8-ceef-40dd-a6ce-d60d2dd34c25\") " Nov 03 00:15:52.047908 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:52.047886 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d26cca8-ceef-40dd-a6ce-d60d2dd34c25-kube-api-access-wrrdz\" (UniqueName: \"kubernetes.io/projected/4d26cca8-ceef-40dd-a6ce-d60d2dd34c25-kube-api-access-wrrdz\") pod \"4d26cca8-ceef-40dd-a6ce-d60d2dd34c25\" (UID: \"4d26cca8-ceef-40dd-a6ce-d60d2dd34c25\") " Nov 03 00:15:52.048104 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:52.047938 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d26cca8-ceef-40dd-a6ce-d60d2dd34c25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d26cca8-ceef-40dd-a6ce-d60d2dd34c25-maas-api-xks-ca\") pod \"4d26cca8-ceef-40dd-a6ce-d60d2dd34c25\" (UID: \"4d26cca8-ceef-40dd-a6ce-d60d2dd34c25\") " Nov 03 00:15:52.048197 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:52.048168 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d26cca8-ceef-40dd-a6ce-d60d2dd34c25-maas-api-openshift-ca" pod "4d26cca8-ceef-40dd-a6ce-d60d2dd34c25" (UID: "4d26cca8-ceef-40dd-a6ce-d60d2dd34c25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:52.048294 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:52.048277 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d26cca8-ceef-40dd-a6ce-d60d2dd34c25-maas-api-openshift-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:52.050236 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:52.050206 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d26cca8-ceef-40dd-a6ce-d60d2dd34c25-kube-api-access-wrrdz" pod "4d26cca8-ceef-40dd-a6ce-d60d2dd34c25" (UID: "4d26cca8-ceef-40dd-a6ce-d60d2dd34c25"). InnerVolumeSpecName "kube-api-access-wrrdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:52.050236 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:52.050219 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d26cca8-ceef-40dd-a6ce-d60d2dd34c25-maas-api-xks-ca" pod "4d26cca8-ceef-40dd-a6ce-d60d2dd34c25" (UID: "4d26cca8-ceef-40dd-a6ce-d60d2dd34c25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:15:52.149608 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:52.149574 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrrdz\" (UniqueName: \"kubernetes.io/projected/4d26cca8-ceef-40dd-a6ce-d60d2dd34c25-kube-api-access-wrrdz\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:52.149608 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:52.149605 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d26cca8-ceef-40dd-a6ce-d60d2dd34c25-maas-api-xks-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:52.805383 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:52.805351 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d31257c-2d46d" event={"ID":"4d26cca8-ceef-40dd-a6ce-d60d2dd34c25","Type":"ContainerDied","Data":"c92a210f50da157d2e20b78d1495f833b03d0fb7f980634017ad0cfad1565df2"} Nov 03 00:15:52.805536 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:52.805408 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c92a210f50da157d2e20b78d1495f833b03d0fb7f980634017ad0cfad1565df2" Nov 03 00:15:52.805536 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:15:52.805364 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d31257c-2d46d" Nov 03 00:15:57.846437 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:15:57.846400 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:16:15.953400 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:15.953363 5727 generic.go:356] "Generic (PLEG): container finished" podID="349fa078-8c82-4c07-a596-900fbb745d7f" containerID="2aa5408d4b51030c955b6fb67b3e6371c269658815b104b5c386157e6e44f36f" exitCode=1 Nov 03 00:16:15.953832 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:15.953437 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x6h9v" event={"ID":"349fa078-8c82-4c07-a596-900fbb745d7f","Type":"ContainerDied","Data":"2aa5408d4b51030c955b6fb67b3e6371c269658815b104b5c386157e6e44f36f"} Nov 03 00:16:15.953832 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:15.953744 5727 scope.go:122] "RemoveContainer" containerID="2aa5408d4b51030c955b6fb67b3e6371c269658815b104b5c386157e6e44f36f" Nov 03 00:16:16.962041 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:16.962004 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x6h9v" event={"ID":"349fa078-8c82-4c07-a596-900fbb745d7f","Type":"ContainerStarted","Data":"dfce53bc02567c35abec8774b8ee03a350860216a5a51742f3ce0f2a6458aaef"} Nov 03 00:16:16.962585 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:16.962317 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x6h9v" Nov 03 00:16:22.777028 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:22.776988 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-3bee1b-5899c6f99f-8ftj2"] Nov 03 00:16:22.882273 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:22.882240 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bee1b-64dd49579f-7psbr"] Nov 03 00:16:27.778034 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:27.777970 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bee1b-5899c6f99f-8ftj2" podUID="940cf7dc-acaa-4ff3-b473-99c7536913cf" containerName="payload-processing" containerID="cri-o://d4d844e36ae03f4a5627ac0c37488838b4c891597114b88e00cff75a8bedf262" gracePeriod=25 Nov 03 00:16:27.844320 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:16:27.844275 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:16:27.887640 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:27.887574 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bee1b-64dd49579f-7psbr" podUID="d6dcd3f9-d170-404b-9921-83b0d2e47ed4" containerName="payload-pre-processing" containerID="cri-o://a891654076a884a2a8fc9804fdcef701771653a431667caea89def289c9d63ee" gracePeriod=25 Nov 03 00:16:27.972340 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:27.972309 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x6h9v" Nov 03 00:16:33.015775 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:33.015741 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bee1b-64dd49579f-7psbr" podUID="d6dcd3f9-d170-404b-9921-83b0d2e47ed4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:16:33.016190 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:33.015746 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bee1b-5899c6f99f-8ftj2" podUID="940cf7dc-acaa-4ff3-b473-99c7536913cf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:16:43.015540 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:43.015496 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bee1b-5899c6f99f-8ftj2" podUID="940cf7dc-acaa-4ff3-b473-99c7536913cf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:16:43.015961 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:43.015512 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bee1b-64dd49579f-7psbr" podUID="d6dcd3f9-d170-404b-9921-83b0d2e47ed4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:16:48.035817 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:48.035745 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bee1b-5899c6f99f-8ftj2" Nov 03 00:16:48.110697 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:48.110667 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/940cf7dc-acaa-4ff3-b473-99c7536913cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/940cf7dc-acaa-4ff3-b473-99c7536913cf-plugins-config-volume\") pod \"940cf7dc-acaa-4ff3-b473-99c7536913cf\" (UID: \"940cf7dc-acaa-4ff3-b473-99c7536913cf\") " Nov 03 00:16:48.110887 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:48.110765 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/940cf7dc-acaa-4ff3-b473-99c7536913cf-kube-api-access-c8txq\" (UniqueName: \"kubernetes.io/projected/940cf7dc-acaa-4ff3-b473-99c7536913cf-kube-api-access-c8txq\") pod \"940cf7dc-acaa-4ff3-b473-99c7536913cf\" (UID: \"940cf7dc-acaa-4ff3-b473-99c7536913cf\") " Nov 03 00:16:48.111031 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:48.111006 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/940cf7dc-acaa-4ff3-b473-99c7536913cf-plugins-config-volume" pod "940cf7dc-acaa-4ff3-b473-99c7536913cf" (UID: "940cf7dc-acaa-4ff3-b473-99c7536913cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:16:48.113107 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:48.113079 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940cf7dc-acaa-4ff3-b473-99c7536913cf-kube-api-access-c8txq" pod "940cf7dc-acaa-4ff3-b473-99c7536913cf" (UID: "940cf7dc-acaa-4ff3-b473-99c7536913cf"). InnerVolumeSpecName "kube-api-access-c8txq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:48.130207 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:48.130187 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bee1b-64dd49579f-7psbr" Nov 03 00:16:48.168397 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:48.168362 5727 generic.go:356] "Generic (PLEG): container finished" podID="940cf7dc-acaa-4ff3-b473-99c7536913cf" containerID="d4d844e36ae03f4a5627ac0c37488838b4c891597114b88e00cff75a8bedf262" exitCode=0 Nov 03 00:16:48.168583 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:48.168421 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bee1b-5899c6f99f-8ftj2" Nov 03 00:16:48.168583 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:48.168448 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bee1b-5899c6f99f-8ftj2" event={"ID":"940cf7dc-acaa-4ff3-b473-99c7536913cf","Type":"ContainerDied","Data":"d4d844e36ae03f4a5627ac0c37488838b4c891597114b88e00cff75a8bedf262"} Nov 03 00:16:48.168583 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:48.168533 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bee1b-5899c6f99f-8ftj2" event={"ID":"940cf7dc-acaa-4ff3-b473-99c7536913cf","Type":"ContainerDied","Data":"62bc16fb099378296e5eddf5e3c6c3204139be052311df3db87e72ac411bde1d"} Nov 03 00:16:48.168583 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:48.168566 5727 scope.go:122] "RemoveContainer" containerID="d4d844e36ae03f4a5627ac0c37488838b4c891597114b88e00cff75a8bedf262" Nov 03 00:16:48.170740 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:48.170688 5727 generic.go:356] "Generic (PLEG): container finished" podID="d6dcd3f9-d170-404b-9921-83b0d2e47ed4" containerID="a891654076a884a2a8fc9804fdcef701771653a431667caea89def289c9d63ee" exitCode=0 Nov 03 00:16:48.170971 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:48.170737 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bee1b-64dd49579f-7psbr" event={"ID":"d6dcd3f9-d170-404b-9921-83b0d2e47ed4","Type":"ContainerDied","Data":"a891654076a884a2a8fc9804fdcef701771653a431667caea89def289c9d63ee"} Nov 03 00:16:48.171097 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:48.171013 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bee1b-64dd49579f-7psbr" event={"ID":"d6dcd3f9-d170-404b-9921-83b0d2e47ed4","Type":"ContainerDied","Data":"34f3add97d1a94e4eb2df68d23bca4f2c38951ba74d8b81a46c74fc6ab24b013"} Nov 03 00:16:48.171097 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:48.170749 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bee1b-64dd49579f-7psbr" Nov 03 00:16:48.180042 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:48.180021 5727 scope.go:122] "RemoveContainer" containerID="d4d844e36ae03f4a5627ac0c37488838b4c891597114b88e00cff75a8bedf262" Nov 03 00:16:48.180518 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:16:48.180489 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4d844e36ae03f4a5627ac0c37488838b4c891597114b88e00cff75a8bedf262\": container with ID starting with d4d844e36ae03f4a5627ac0c37488838b4c891597114b88e00cff75a8bedf262 not found: ID does not exist" containerID="d4d844e36ae03f4a5627ac0c37488838b4c891597114b88e00cff75a8bedf262" Nov 03 00:16:48.180604 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:48.180532 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4d844e36ae03f4a5627ac0c37488838b4c891597114b88e00cff75a8bedf262"} err="failed to get container status \"d4d844e36ae03f4a5627ac0c37488838b4c891597114b88e00cff75a8bedf262\": rpc error: code = NotFound desc = could not find container \"d4d844e36ae03f4a5627ac0c37488838b4c891597114b88e00cff75a8bedf262\": container with ID starting with d4d844e36ae03f4a5627ac0c37488838b4c891597114b88e00cff75a8bedf262 not found: ID does not exist" Nov 03 00:16:48.180604 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:48.180559 5727 scope.go:122] "RemoveContainer" containerID="a891654076a884a2a8fc9804fdcef701771653a431667caea89def289c9d63ee" Nov 03 00:16:48.192309 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:48.192290 5727 scope.go:122] "RemoveContainer" containerID="a891654076a884a2a8fc9804fdcef701771653a431667caea89def289c9d63ee" Nov 03 00:16:48.192567 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:16:48.192544 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a891654076a884a2a8fc9804fdcef701771653a431667caea89def289c9d63ee\": container with ID starting with a891654076a884a2a8fc9804fdcef701771653a431667caea89def289c9d63ee not found: ID does not exist" containerID="a891654076a884a2a8fc9804fdcef701771653a431667caea89def289c9d63ee" Nov 03 00:16:48.192643 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:48.192579 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a891654076a884a2a8fc9804fdcef701771653a431667caea89def289c9d63ee"} err="failed to get container status \"a891654076a884a2a8fc9804fdcef701771653a431667caea89def289c9d63ee\": rpc error: code = NotFound desc = could not find container \"a891654076a884a2a8fc9804fdcef701771653a431667caea89def289c9d63ee\": container with ID starting with a891654076a884a2a8fc9804fdcef701771653a431667caea89def289c9d63ee not found: ID does not exist" Nov 03 00:16:48.212358 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:48.212332 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6dcd3f9-d170-404b-9921-83b0d2e47ed4-kube-api-access-btm5x\" (UniqueName: \"kubernetes.io/projected/d6dcd3f9-d170-404b-9921-83b0d2e47ed4-kube-api-access-btm5x\") pod \"d6dcd3f9-d170-404b-9921-83b0d2e47ed4\" (UID: \"d6dcd3f9-d170-404b-9921-83b0d2e47ed4\") " Nov 03 00:16:48.212455 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:48.212444 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6dcd3f9-d170-404b-9921-83b0d2e47ed4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6dcd3f9-d170-404b-9921-83b0d2e47ed4-plugins-config-volume\") pod \"d6dcd3f9-d170-404b-9921-83b0d2e47ed4\" (UID: \"d6dcd3f9-d170-404b-9921-83b0d2e47ed4\") " Nov 03 00:16:48.212790 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:48.212762 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8txq\" (UniqueName: \"kubernetes.io/projected/940cf7dc-acaa-4ff3-b473-99c7536913cf-kube-api-access-c8txq\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:48.212889 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:48.212794 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/940cf7dc-acaa-4ff3-b473-99c7536913cf-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:48.212889 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:48.212853 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6dcd3f9-d170-404b-9921-83b0d2e47ed4-plugins-config-volume" pod "d6dcd3f9-d170-404b-9921-83b0d2e47ed4" (UID: "d6dcd3f9-d170-404b-9921-83b0d2e47ed4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:16:48.214595 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:48.214575 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6dcd3f9-d170-404b-9921-83b0d2e47ed4-kube-api-access-btm5x" pod "d6dcd3f9-d170-404b-9921-83b0d2e47ed4" (UID: "d6dcd3f9-d170-404b-9921-83b0d2e47ed4"). InnerVolumeSpecName "kube-api-access-btm5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:48.314344 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:48.314320 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btm5x\" (UniqueName: \"kubernetes.io/projected/d6dcd3f9-d170-404b-9921-83b0d2e47ed4-kube-api-access-btm5x\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:48.314508 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:48.314353 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6dcd3f9-d170-404b-9921-83b0d2e47ed4-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:48.429976 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:48.429939 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-3bee1b-5899c6f99f-8ftj2"] Nov 03 00:16:48.433969 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:48.433942 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-3bee1b-5899c6f99f-8ftj2"] Nov 03 00:16:48.591017 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:48.590950 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bee1b-64dd49579f-7psbr"] Nov 03 00:16:48.594547 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:48.594526 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bee1b-64dd49579f-7psbr"] Nov 03 00:16:49.883696 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:49.883658 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="940cf7dc-acaa-4ff3-b473-99c7536913cf" path="/var/lib/kubelet/pods/940cf7dc-acaa-4ff3-b473-99c7536913cf/volumes" Nov 03 00:16:49.884228 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:49.884045 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6dcd3f9-d170-404b-9921-83b0d2e47ed4" path="/var/lib/kubelet/pods/d6dcd3f9-d170-404b-9921-83b0d2e47ed4/volumes" Nov 03 00:16:51.863679 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:51.863648 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9b6c9e-54cc6f6cdf-dr7kw"] Nov 03 00:16:51.971635 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:51.971595 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9b6c9e-6d4bc8b6f9-dztdx"] Nov 03 00:16:56.864814 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:56.864731 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9b6c9e-54cc6f6cdf-dr7kw" podUID="49421404-0145-4fda-a9a4-26c7ba952374" containerName="payload-processing" containerID="cri-o://cbf3cf4e32644d550ca821dd60c42ade5f93b3edc2f6cd3023cd093ff1efcb56" gracePeriod=25 Nov 03 00:16:56.972065 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:16:56.972030 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9b6c9e-6d4bc8b6f9-dztdx" podUID="4bd218d6-1000-4fce-bade-c54fa8e7026b" containerName="payload-pre-processing" containerID="cri-o://981c3089f1f5c517bfe95b024ea43a5d483029d67cfdea62393a918ae4bc4e1d" gracePeriod=25 Nov 03 00:16:57.846120 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:16:57.846076 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:17:02.288223 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:02.288180 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9b6c9e-54cc6f6cdf-dr7kw" podUID="49421404-0145-4fda-a9a4-26c7ba952374" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:17:02.288639 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:02.288256 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9b6c9e-6d4bc8b6f9-dztdx" podUID="4bd218d6-1000-4fce-bade-c54fa8e7026b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:17:10.454388 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:10.454351 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q"] Nov 03 00:17:10.504295 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:10.504264 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5"] Nov 03 00:17:12.287907 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:12.287875 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9b6c9e-6d4bc8b6f9-dztdx" podUID="4bd218d6-1000-4fce-bade-c54fa8e7026b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:17:12.288353 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:12.287949 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9b6c9e-54cc6f6cdf-dr7kw" podUID="49421404-0145-4fda-a9a4-26c7ba952374" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:17:17.134466 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.134395 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9b6c9e-54cc6f6cdf-dr7kw" Nov 03 00:17:17.192785 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.192750 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49421404-0145-4fda-a9a4-26c7ba952374-kube-api-access-vq47q\" (UniqueName: \"kubernetes.io/projected/49421404-0145-4fda-a9a4-26c7ba952374-kube-api-access-vq47q\") pod \"49421404-0145-4fda-a9a4-26c7ba952374\" (UID: \"49421404-0145-4fda-a9a4-26c7ba952374\") " Nov 03 00:17:17.192971 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.192871 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49421404-0145-4fda-a9a4-26c7ba952374-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49421404-0145-4fda-a9a4-26c7ba952374-plugins-config-volume\") pod \"49421404-0145-4fda-a9a4-26c7ba952374\" (UID: \"49421404-0145-4fda-a9a4-26c7ba952374\") " Nov 03 00:17:17.193322 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.193289 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49421404-0145-4fda-a9a4-26c7ba952374-plugins-config-volume" pod "49421404-0145-4fda-a9a4-26c7ba952374" (UID: "49421404-0145-4fda-a9a4-26c7ba952374"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:17:17.195664 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.195637 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49421404-0145-4fda-a9a4-26c7ba952374-kube-api-access-vq47q" pod "49421404-0145-4fda-a9a4-26c7ba952374" (UID: "49421404-0145-4fda-a9a4-26c7ba952374"). InnerVolumeSpecName "kube-api-access-vq47q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:17.224924 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.224901 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9b6c9e-6d4bc8b6f9-dztdx" Nov 03 00:17:17.294010 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.293985 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bd218d6-1000-4fce-bade-c54fa8e7026b-kube-api-access-f9cht\" (UniqueName: \"kubernetes.io/projected/4bd218d6-1000-4fce-bade-c54fa8e7026b-kube-api-access-f9cht\") pod \"4bd218d6-1000-4fce-bade-c54fa8e7026b\" (UID: \"4bd218d6-1000-4fce-bade-c54fa8e7026b\") " Nov 03 00:17:17.294144 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.294035 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bd218d6-1000-4fce-bade-c54fa8e7026b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd218d6-1000-4fce-bade-c54fa8e7026b-plugins-config-volume\") pod \"4bd218d6-1000-4fce-bade-c54fa8e7026b\" (UID: \"4bd218d6-1000-4fce-bade-c54fa8e7026b\") " Nov 03 00:17:17.294313 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.294293 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq47q\" (UniqueName: \"kubernetes.io/projected/49421404-0145-4fda-a9a4-26c7ba952374-kube-api-access-vq47q\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:17.294354 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.294320 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49421404-0145-4fda-a9a4-26c7ba952374-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:17.294467 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.294451 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bd218d6-1000-4fce-bade-c54fa8e7026b-plugins-config-volume" pod "4bd218d6-1000-4fce-bade-c54fa8e7026b" (UID: "4bd218d6-1000-4fce-bade-c54fa8e7026b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:17:17.296185 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.296165 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bd218d6-1000-4fce-bade-c54fa8e7026b-kube-api-access-f9cht" pod "4bd218d6-1000-4fce-bade-c54fa8e7026b" (UID: "4bd218d6-1000-4fce-bade-c54fa8e7026b"). InnerVolumeSpecName "kube-api-access-f9cht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:17.358228 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.358190 5727 generic.go:356] "Generic (PLEG): container finished" podID="49421404-0145-4fda-a9a4-26c7ba952374" containerID="cbf3cf4e32644d550ca821dd60c42ade5f93b3edc2f6cd3023cd093ff1efcb56" exitCode=0 Nov 03 00:17:17.358394 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.358254 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9b6c9e-54cc6f6cdf-dr7kw" Nov 03 00:17:17.358394 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.358280 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9b6c9e-54cc6f6cdf-dr7kw" event={"ID":"49421404-0145-4fda-a9a4-26c7ba952374","Type":"ContainerDied","Data":"cbf3cf4e32644d550ca821dd60c42ade5f93b3edc2f6cd3023cd093ff1efcb56"} Nov 03 00:17:17.358394 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.358350 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9b6c9e-54cc6f6cdf-dr7kw" event={"ID":"49421404-0145-4fda-a9a4-26c7ba952374","Type":"ContainerDied","Data":"aca9ee27946491e91e1babdd672aed621162ec5a2615346a62ff5c9f536cf0bc"} Nov 03 00:17:17.358394 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.358378 5727 scope.go:122] "RemoveContainer" containerID="cbf3cf4e32644d550ca821dd60c42ade5f93b3edc2f6cd3023cd093ff1efcb56" Nov 03 00:17:17.362301 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.362187 5727 generic.go:356] "Generic (PLEG): container finished" podID="4bd218d6-1000-4fce-bade-c54fa8e7026b" containerID="981c3089f1f5c517bfe95b024ea43a5d483029d67cfdea62393a918ae4bc4e1d" exitCode=0 Nov 03 00:17:17.362301 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.362233 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9b6c9e-6d4bc8b6f9-dztdx" event={"ID":"4bd218d6-1000-4fce-bade-c54fa8e7026b","Type":"ContainerDied","Data":"981c3089f1f5c517bfe95b024ea43a5d483029d67cfdea62393a918ae4bc4e1d"} Nov 03 00:17:17.362301 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.362264 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9b6c9e-6d4bc8b6f9-dztdx" Nov 03 00:17:17.362301 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.362272 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9b6c9e-6d4bc8b6f9-dztdx" event={"ID":"4bd218d6-1000-4fce-bade-c54fa8e7026b","Type":"ContainerDied","Data":"2b9eb6441452397850a52c032aa121c73182a7aba948a409c48fe0f124a9fd89"} Nov 03 00:17:17.376941 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.376917 5727 scope.go:122] "RemoveContainer" containerID="cbf3cf4e32644d550ca821dd60c42ade5f93b3edc2f6cd3023cd093ff1efcb56" Nov 03 00:17:17.377292 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:17:17.377245 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbf3cf4e32644d550ca821dd60c42ade5f93b3edc2f6cd3023cd093ff1efcb56\": container with ID starting with cbf3cf4e32644d550ca821dd60c42ade5f93b3edc2f6cd3023cd093ff1efcb56 not found: ID does not exist" containerID="cbf3cf4e32644d550ca821dd60c42ade5f93b3edc2f6cd3023cd093ff1efcb56" Nov 03 00:17:17.377379 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.377306 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf3cf4e32644d550ca821dd60c42ade5f93b3edc2f6cd3023cd093ff1efcb56"} err="failed to get container status \"cbf3cf4e32644d550ca821dd60c42ade5f93b3edc2f6cd3023cd093ff1efcb56\": rpc error: code = NotFound desc = could not find container \"cbf3cf4e32644d550ca821dd60c42ade5f93b3edc2f6cd3023cd093ff1efcb56\": container with ID starting with cbf3cf4e32644d550ca821dd60c42ade5f93b3edc2f6cd3023cd093ff1efcb56 not found: ID does not exist" Nov 03 00:17:17.377379 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.377331 5727 scope.go:122] "RemoveContainer" containerID="981c3089f1f5c517bfe95b024ea43a5d483029d67cfdea62393a918ae4bc4e1d" Nov 03 00:17:17.388453 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.388435 5727 scope.go:122] "RemoveContainer" containerID="981c3089f1f5c517bfe95b024ea43a5d483029d67cfdea62393a918ae4bc4e1d" Nov 03 00:17:17.388758 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:17:17.388738 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"981c3089f1f5c517bfe95b024ea43a5d483029d67cfdea62393a918ae4bc4e1d\": container with ID starting with 981c3089f1f5c517bfe95b024ea43a5d483029d67cfdea62393a918ae4bc4e1d not found: ID does not exist" containerID="981c3089f1f5c517bfe95b024ea43a5d483029d67cfdea62393a918ae4bc4e1d" Nov 03 00:17:17.388822 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.388765 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"981c3089f1f5c517bfe95b024ea43a5d483029d67cfdea62393a918ae4bc4e1d"} err="failed to get container status \"981c3089f1f5c517bfe95b024ea43a5d483029d67cfdea62393a918ae4bc4e1d\": rpc error: code = NotFound desc = could not find container \"981c3089f1f5c517bfe95b024ea43a5d483029d67cfdea62393a918ae4bc4e1d\": container with ID starting with 981c3089f1f5c517bfe95b024ea43a5d483029d67cfdea62393a918ae4bc4e1d not found: ID does not exist" Nov 03 00:17:17.395404 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.395377 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9cht\" (UniqueName: \"kubernetes.io/projected/4bd218d6-1000-4fce-bade-c54fa8e7026b-kube-api-access-f9cht\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:17.395513 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.395409 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd218d6-1000-4fce-bade-c54fa8e7026b-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:17.580305 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.580264 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9b6c9e-6d4bc8b6f9-dztdx"] Nov 03 00:17:17.585500 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.585464 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9b6c9e-6d4bc8b6f9-dztdx"] Nov 03 00:17:17.695617 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.695530 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9b6c9e-54cc6f6cdf-dr7kw"] Nov 03 00:17:17.699516 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.699484 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9b6c9e-54cc6f6cdf-dr7kw"] Nov 03 00:17:17.884094 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.884054 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49421404-0145-4fda-a9a4-26c7ba952374" path="/var/lib/kubelet/pods/49421404-0145-4fda-a9a4-26c7ba952374/volumes" Nov 03 00:17:17.884560 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:17.884536 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bd218d6-1000-4fce-bade-c54fa8e7026b" path="/var/lib/kubelet/pods/4bd218d6-1000-4fce-bade-c54fa8e7026b/volumes" Nov 03 00:17:19.218088 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:19.218048 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b42983-f5cc57d49-hbnxp"] Nov 03 00:17:24.218747 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:24.218700 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-b42983-f5cc57d49-hbnxp" podUID="3472bbbd-3579-4f37-a8f0-48fcbe1cb627" containerName="payload-processing" containerID="cri-o://3a5aacdce3f729603ead20ef22bf908a8fa99baf403cd81d70c2fbb9b1499b14" gracePeriod=25 Nov 03 00:17:25.476279 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.476229 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" podUID="d3799f21-d78b-468d-9347-ca3f0968409d" containerName="main" containerID="cri-o://0b7a6bb7b7d8fa768146f7a64038da011c55643970d36d0a2789a74b691349e1" gracePeriod=45 Nov 03 00:17:25.525220 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.525176 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" podUID="f83c8651-5db1-4e80-b9fb-ec5b793e7e94" containerName="main" containerID="cri-o://190e9d3ee7b477fdb9ac882f06e6b972dce843fcc7bdcfb0048fceb1ff225b72" gracePeriod=45 Nov 03 00:17:25.809057 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.809027 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" Nov 03 00:17:25.833737 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.833711 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" Nov 03 00:17:25.880761 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.880729 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-dshm\") pod \"f83c8651-5db1-4e80-b9fb-ec5b793e7e94\" (UID: \"f83c8651-5db1-4e80-b9fb-ec5b793e7e94\") " Nov 03 00:17:25.880948 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.880784 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-kube-api-access-fnvxc\" (UniqueName: \"kubernetes.io/projected/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-kube-api-access-fnvxc\") pod \"f83c8651-5db1-4e80-b9fb-ec5b793e7e94\" (UID: \"f83c8651-5db1-4e80-b9fb-ec5b793e7e94\") " Nov 03 00:17:25.880948 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.880839 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-home\" (UniqueName: \"kubernetes.io/empty-dir/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-home\") pod \"f83c8651-5db1-4e80-b9fb-ec5b793e7e94\" (UID: \"f83c8651-5db1-4e80-b9fb-ec5b793e7e94\") " Nov 03 00:17:25.880948 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.880873 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3799f21-d78b-468d-9347-ca3f0968409d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3799f21-d78b-468d-9347-ca3f0968409d-tmp-dir\") pod \"d3799f21-d78b-468d-9347-ca3f0968409d\" (UID: \"d3799f21-d78b-468d-9347-ca3f0968409d\") " Nov 03 00:17:25.880948 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.880925 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3799f21-d78b-468d-9347-ca3f0968409d-home\" (UniqueName: \"kubernetes.io/empty-dir/d3799f21-d78b-468d-9347-ca3f0968409d-home\") pod \"d3799f21-d78b-468d-9347-ca3f0968409d\" (UID: \"d3799f21-d78b-468d-9347-ca3f0968409d\") " Nov 03 00:17:25.881249 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.880956 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-tls-certs\" (UniqueName: \"kubernetes.io/secret/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-tls-certs\") pod \"f83c8651-5db1-4e80-b9fb-ec5b793e7e94\" (UID: \"f83c8651-5db1-4e80-b9fb-ec5b793e7e94\") " Nov 03 00:17:25.881249 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.880983 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3799f21-d78b-468d-9347-ca3f0968409d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3799f21-d78b-468d-9347-ca3f0968409d-model-cache\") pod \"d3799f21-d78b-468d-9347-ca3f0968409d\" (UID: \"d3799f21-d78b-468d-9347-ca3f0968409d\") " Nov 03 00:17:25.881249 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.881023 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3799f21-d78b-468d-9347-ca3f0968409d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3799f21-d78b-468d-9347-ca3f0968409d-dshm\") pod \"d3799f21-d78b-468d-9347-ca3f0968409d\" (UID: \"d3799f21-d78b-468d-9347-ca3f0968409d\") " Nov 03 00:17:25.881249 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.881051 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3799f21-d78b-468d-9347-ca3f0968409d-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3799f21-d78b-468d-9347-ca3f0968409d-tls-certs\") pod \"d3799f21-d78b-468d-9347-ca3f0968409d\" (UID: \"d3799f21-d78b-468d-9347-ca3f0968409d\") " Nov 03 00:17:25.881249 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.881097 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3799f21-d78b-468d-9347-ca3f0968409d-kube-api-access-md42w\" (UniqueName: \"kubernetes.io/projected/d3799f21-d78b-468d-9347-ca3f0968409d-kube-api-access-md42w\") pod \"d3799f21-d78b-468d-9347-ca3f0968409d\" (UID: \"d3799f21-d78b-468d-9347-ca3f0968409d\") " Nov 03 00:17:25.881249 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.881172 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-tmp-dir\") pod \"f83c8651-5db1-4e80-b9fb-ec5b793e7e94\" (UID: \"f83c8651-5db1-4e80-b9fb-ec5b793e7e94\") " Nov 03 00:17:25.881249 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.881210 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-model-cache\") pod \"f83c8651-5db1-4e80-b9fb-ec5b793e7e94\" (UID: \"f83c8651-5db1-4e80-b9fb-ec5b793e7e94\") " Nov 03 00:17:25.881790 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.881409 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-home" pod "f83c8651-5db1-4e80-b9fb-ec5b793e7e94" (UID: "f83c8651-5db1-4e80-b9fb-ec5b793e7e94"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:25.881790 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.881675 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-model-cache" pod "f83c8651-5db1-4e80-b9fb-ec5b793e7e94" (UID: "f83c8651-5db1-4e80-b9fb-ec5b793e7e94"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:25.881790 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.881716 5727 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-home\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:25.882106 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.882079 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3799f21-d78b-468d-9347-ca3f0968409d-model-cache" pod "d3799f21-d78b-468d-9347-ca3f0968409d" (UID: "d3799f21-d78b-468d-9347-ca3f0968409d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:25.882343 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.882323 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-tmp-dir" pod "f83c8651-5db1-4e80-b9fb-ec5b793e7e94" (UID: "f83c8651-5db1-4e80-b9fb-ec5b793e7e94"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:25.882721 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.882698 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3799f21-d78b-468d-9347-ca3f0968409d-home" pod "d3799f21-d78b-468d-9347-ca3f0968409d" (UID: "d3799f21-d78b-468d-9347-ca3f0968409d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:25.882801 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.882769 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3799f21-d78b-468d-9347-ca3f0968409d-tmp-dir" pod "d3799f21-d78b-468d-9347-ca3f0968409d" (UID: "d3799f21-d78b-468d-9347-ca3f0968409d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:25.883765 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.883737 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-dshm" pod "f83c8651-5db1-4e80-b9fb-ec5b793e7e94" (UID: "f83c8651-5db1-4e80-b9fb-ec5b793e7e94"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:25.883878 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.883821 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3799f21-d78b-468d-9347-ca3f0968409d-dshm" pod "d3799f21-d78b-468d-9347-ca3f0968409d" (UID: "d3799f21-d78b-468d-9347-ca3f0968409d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:25.884245 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.884216 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3799f21-d78b-468d-9347-ca3f0968409d-tls-certs" pod "d3799f21-d78b-468d-9347-ca3f0968409d" (UID: "d3799f21-d78b-468d-9347-ca3f0968409d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:17:25.884500 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.884464 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3799f21-d78b-468d-9347-ca3f0968409d-kube-api-access-md42w" pod "d3799f21-d78b-468d-9347-ca3f0968409d" (UID: "d3799f21-d78b-468d-9347-ca3f0968409d"). InnerVolumeSpecName "kube-api-access-md42w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:25.884631 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.884604 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-kube-api-access-fnvxc" pod "f83c8651-5db1-4e80-b9fb-ec5b793e7e94" (UID: "f83c8651-5db1-4e80-b9fb-ec5b793e7e94"). InnerVolumeSpecName "kube-api-access-fnvxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:25.885274 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.885247 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-tls-certs" pod "f83c8651-5db1-4e80-b9fb-ec5b793e7e94" (UID: "f83c8651-5db1-4e80-b9fb-ec5b793e7e94"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:17:25.982910 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.982731 5727 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-tmp-dir\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:25.982910 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.982763 5727 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-model-cache\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:25.982910 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.982781 5727 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-dshm\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:25.982910 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.982797 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnvxc\" (UniqueName: \"kubernetes.io/projected/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-kube-api-access-fnvxc\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:25.982910 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.982814 5727 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3799f21-d78b-468d-9347-ca3f0968409d-tmp-dir\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:25.982910 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.982827 5727 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3799f21-d78b-468d-9347-ca3f0968409d-home\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:25.982910 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.982840 5727 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f83c8651-5db1-4e80-b9fb-ec5b793e7e94-tls-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:25.982910 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.982855 5727 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3799f21-d78b-468d-9347-ca3f0968409d-model-cache\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:25.982910 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.982865 5727 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3799f21-d78b-468d-9347-ca3f0968409d-dshm\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:25.982910 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.982873 5727 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3799f21-d78b-468d-9347-ca3f0968409d-tls-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:25.982910 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:25.982881 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md42w\" (UniqueName: \"kubernetes.io/projected/d3799f21-d78b-468d-9347-ca3f0968409d-kube-api-access-md42w\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:26.425258 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:26.425218 5727 generic.go:356] "Generic (PLEG): container finished" podID="d3799f21-d78b-468d-9347-ca3f0968409d" containerID="0b7a6bb7b7d8fa768146f7a64038da011c55643970d36d0a2789a74b691349e1" exitCode=0 Nov 03 00:17:26.425258 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:26.425260 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" Nov 03 00:17:26.425535 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:26.425297 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" event={"ID":"d3799f21-d78b-468d-9347-ca3f0968409d","Type":"ContainerDied","Data":"0b7a6bb7b7d8fa768146f7a64038da011c55643970d36d0a2789a74b691349e1"} Nov 03 00:17:26.425535 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:26.425345 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q" event={"ID":"d3799f21-d78b-468d-9347-ca3f0968409d","Type":"ContainerDied","Data":"f8040b5e4a0eb96cd9505aa72a919f0f8dad9c7904784b924954e55a23d31b2e"} Nov 03 00:17:26.425535 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:26.425368 5727 scope.go:122] "RemoveContainer" containerID="0b7a6bb7b7d8fa768146f7a64038da011c55643970d36d0a2789a74b691349e1" Nov 03 00:17:26.426940 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:26.426912 5727 generic.go:356] "Generic (PLEG): container finished" podID="f83c8651-5db1-4e80-b9fb-ec5b793e7e94" containerID="190e9d3ee7b477fdb9ac882f06e6b972dce843fcc7bdcfb0048fceb1ff225b72" exitCode=0 Nov 03 00:17:26.427100 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:26.426947 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" Nov 03 00:17:26.427100 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:26.426980 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" event={"ID":"f83c8651-5db1-4e80-b9fb-ec5b793e7e94","Type":"ContainerDied","Data":"190e9d3ee7b477fdb9ac882f06e6b972dce843fcc7bdcfb0048fceb1ff225b72"} Nov 03 00:17:26.427100 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:26.427051 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5" event={"ID":"f83c8651-5db1-4e80-b9fb-ec5b793e7e94","Type":"ContainerDied","Data":"ba123b3cd1f77756b3ab8074d90a929a3b67369cceaec80003672e5343474fb8"} Nov 03 00:17:26.437410 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:26.437391 5727 scope.go:122] "RemoveContainer" containerID="0b7a6bb7b7d8fa768146f7a64038da011c55643970d36d0a2789a74b691349e1" Nov 03 00:17:26.437675 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:17:26.437656 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b7a6bb7b7d8fa768146f7a64038da011c55643970d36d0a2789a74b691349e1\": container with ID starting with 0b7a6bb7b7d8fa768146f7a64038da011c55643970d36d0a2789a74b691349e1 not found: ID does not exist" containerID="0b7a6bb7b7d8fa768146f7a64038da011c55643970d36d0a2789a74b691349e1" Nov 03 00:17:26.437757 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:26.437686 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b7a6bb7b7d8fa768146f7a64038da011c55643970d36d0a2789a74b691349e1"} err="failed to get container status \"0b7a6bb7b7d8fa768146f7a64038da011c55643970d36d0a2789a74b691349e1\": rpc error: code = NotFound desc = could not find container \"0b7a6bb7b7d8fa768146f7a64038da011c55643970d36d0a2789a74b691349e1\": container with ID starting with 0b7a6bb7b7d8fa768146f7a64038da011c55643970d36d0a2789a74b691349e1 not found: ID does not exist" Nov 03 00:17:26.437757 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:26.437708 5727 scope.go:122] "RemoveContainer" containerID="190e9d3ee7b477fdb9ac882f06e6b972dce843fcc7bdcfb0048fceb1ff225b72" Nov 03 00:17:26.446998 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:26.446976 5727 scope.go:122] "RemoveContainer" containerID="190e9d3ee7b477fdb9ac882f06e6b972dce843fcc7bdcfb0048fceb1ff225b72" Nov 03 00:17:26.447328 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:17:26.447307 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"190e9d3ee7b477fdb9ac882f06e6b972dce843fcc7bdcfb0048fceb1ff225b72\": container with ID starting with 190e9d3ee7b477fdb9ac882f06e6b972dce843fcc7bdcfb0048fceb1ff225b72 not found: ID does not exist" containerID="190e9d3ee7b477fdb9ac882f06e6b972dce843fcc7bdcfb0048fceb1ff225b72" Nov 03 00:17:26.447418 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:26.447333 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"190e9d3ee7b477fdb9ac882f06e6b972dce843fcc7bdcfb0048fceb1ff225b72"} err="failed to get container status \"190e9d3ee7b477fdb9ac882f06e6b972dce843fcc7bdcfb0048fceb1ff225b72\": rpc error: code = NotFound desc = could not find container \"190e9d3ee7b477fdb9ac882f06e6b972dce843fcc7bdcfb0048fceb1ff225b72\": container with ID starting with 190e9d3ee7b477fdb9ac882f06e6b972dce843fcc7bdcfb0048fceb1ff225b72 not found: ID does not exist" Nov 03 00:17:26.627154 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:26.626638 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q"] Nov 03 00:17:26.693879 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:26.683572 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b42983/distinct-2-w1-b42983-kserve-b65669645-72g6q"] Nov 03 00:17:26.788275 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:26.788234 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5"] Nov 03 00:17:26.793510 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:26.793483 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b42983/premium-w1-b42983-kserve-865c76d74c-clnj5"] Nov 03 00:17:26.925009 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:26.924975 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-b42983-f5cc57d49-hbnxp" podUID="3472bbbd-3579-4f37-a8f0-48fcbe1cb627" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:17:27.845708 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:17:27.845670 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:17:27.881272 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:27.881245 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3799f21-d78b-468d-9347-ca3f0968409d" path="/var/lib/kubelet/pods/d3799f21-d78b-468d-9347-ca3f0968409d/volumes" Nov 03 00:17:27.881667 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:27.881653 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f83c8651-5db1-4e80-b9fb-ec5b793e7e94" path="/var/lib/kubelet/pods/f83c8651-5db1-4e80-b9fb-ec5b793e7e94/volumes" Nov 03 00:17:30.669348 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:30.669245 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc"] Nov 03 00:17:30.669831 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:30.669505 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc" podUID="13206437-6c0e-4cbe-9f87-ce696dc65d6f" containerName="maas-api" containerID="cri-o://83fcb42d8e9bd3c1882e84987a4b85dff9af8ed08444fab11545fed9a19727ec" gracePeriod=30 Nov 03 00:17:30.929775 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:30.929718 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc" Nov 03 00:17:31.039039 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:31.039007 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13206437-6c0e-4cbe-9f87-ce696dc65d6f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13206437-6c0e-4cbe-9f87-ce696dc65d6f-openshift-service-ca\") pod \"13206437-6c0e-4cbe-9f87-ce696dc65d6f\" (UID: \"13206437-6c0e-4cbe-9f87-ce696dc65d6f\") " Nov 03 00:17:31.039391 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:31.039066 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13206437-6c0e-4cbe-9f87-ce696dc65d6f-kube-api-access-jcnqp\" (UniqueName: \"kubernetes.io/projected/13206437-6c0e-4cbe-9f87-ce696dc65d6f-kube-api-access-jcnqp\") pod \"13206437-6c0e-4cbe-9f87-ce696dc65d6f\" (UID: \"13206437-6c0e-4cbe-9f87-ce696dc65d6f\") " Nov 03 00:17:31.039391 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:31.039181 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13206437-6c0e-4cbe-9f87-ce696dc65d6f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/13206437-6c0e-4cbe-9f87-ce696dc65d6f-metrics-certs\") pod \"13206437-6c0e-4cbe-9f87-ce696dc65d6f\" (UID: \"13206437-6c0e-4cbe-9f87-ce696dc65d6f\") " Nov 03 00:17:31.039391 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:31.039209 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13206437-6c0e-4cbe-9f87-ce696dc65d6f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13206437-6c0e-4cbe-9f87-ce696dc65d6f-maas-api-tls\") pod \"13206437-6c0e-4cbe-9f87-ce696dc65d6f\" (UID: \"13206437-6c0e-4cbe-9f87-ce696dc65d6f\") " Nov 03 00:17:31.039621 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:31.039544 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13206437-6c0e-4cbe-9f87-ce696dc65d6f-openshift-service-ca" pod "13206437-6c0e-4cbe-9f87-ce696dc65d6f" (UID: "13206437-6c0e-4cbe-9f87-ce696dc65d6f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:17:31.039743 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:31.039722 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13206437-6c0e-4cbe-9f87-ce696dc65d6f-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:31.042095 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:31.042065 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13206437-6c0e-4cbe-9f87-ce696dc65d6f-kube-api-access-jcnqp" pod "13206437-6c0e-4cbe-9f87-ce696dc65d6f" (UID: "13206437-6c0e-4cbe-9f87-ce696dc65d6f"). InnerVolumeSpecName "kube-api-access-jcnqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:31.042274 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:31.042255 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13206437-6c0e-4cbe-9f87-ce696dc65d6f-maas-api-tls" pod "13206437-6c0e-4cbe-9f87-ce696dc65d6f" (UID: "13206437-6c0e-4cbe-9f87-ce696dc65d6f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:17:31.042346 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:31.042308 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13206437-6c0e-4cbe-9f87-ce696dc65d6f-metrics-certs" pod "13206437-6c0e-4cbe-9f87-ce696dc65d6f" (UID: "13206437-6c0e-4cbe-9f87-ce696dc65d6f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:17:31.140791 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:31.140750 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcnqp\" (UniqueName: \"kubernetes.io/projected/13206437-6c0e-4cbe-9f87-ce696dc65d6f-kube-api-access-jcnqp\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:31.140995 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:31.140784 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13206437-6c0e-4cbe-9f87-ce696dc65d6f-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:31.140995 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:31.140830 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13206437-6c0e-4cbe-9f87-ce696dc65d6f-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:31.461705 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:31.461671 5727 generic.go:356] "Generic (PLEG): container finished" podID="13206437-6c0e-4cbe-9f87-ce696dc65d6f" containerID="83fcb42d8e9bd3c1882e84987a4b85dff9af8ed08444fab11545fed9a19727ec" exitCode=0 Nov 03 00:17:31.461896 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:31.461722 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc" Nov 03 00:17:31.461896 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:31.461723 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc" event={"ID":"13206437-6c0e-4cbe-9f87-ce696dc65d6f","Type":"ContainerDied","Data":"83fcb42d8e9bd3c1882e84987a4b85dff9af8ed08444fab11545fed9a19727ec"} Nov 03 00:17:31.461896 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:31.461865 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc" event={"ID":"13206437-6c0e-4cbe-9f87-ce696dc65d6f","Type":"ContainerDied","Data":"187cdbfaacf7f3191c2a1913e508c310ad37fd50f5c56127ba46ee2dbf8c448c"} Nov 03 00:17:31.462041 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:31.461903 5727 scope.go:122] "RemoveContainer" containerID="83fcb42d8e9bd3c1882e84987a4b85dff9af8ed08444fab11545fed9a19727ec" Nov 03 00:17:31.474635 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:31.474617 5727 scope.go:122] "RemoveContainer" containerID="83fcb42d8e9bd3c1882e84987a4b85dff9af8ed08444fab11545fed9a19727ec" Nov 03 00:17:31.474905 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:17:31.474882 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83fcb42d8e9bd3c1882e84987a4b85dff9af8ed08444fab11545fed9a19727ec\": container with ID starting with 83fcb42d8e9bd3c1882e84987a4b85dff9af8ed08444fab11545fed9a19727ec not found: ID does not exist" containerID="83fcb42d8e9bd3c1882e84987a4b85dff9af8ed08444fab11545fed9a19727ec" Nov 03 00:17:31.474949 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:31.474914 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83fcb42d8e9bd3c1882e84987a4b85dff9af8ed08444fab11545fed9a19727ec"} err="failed to get container status \"83fcb42d8e9bd3c1882e84987a4b85dff9af8ed08444fab11545fed9a19727ec\": rpc error: code = NotFound desc = could not find container \"83fcb42d8e9bd3c1882e84987a4b85dff9af8ed08444fab11545fed9a19727ec\": container with ID starting with 83fcb42d8e9bd3c1882e84987a4b85dff9af8ed08444fab11545fed9a19727ec not found: ID does not exist" Nov 03 00:17:31.625458 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:31.625416 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc"] Nov 03 00:17:31.629276 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:31.629251 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42983-769456cb4b-7xrhc"] Nov 03 00:17:31.881414 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:31.881379 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13206437-6c0e-4cbe-9f87-ce696dc65d6f" path="/var/lib/kubelet/pods/13206437-6c0e-4cbe-9f87-ce696dc65d6f/volumes" Nov 03 00:17:36.925317 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:36.925278 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-b42983-f5cc57d49-hbnxp" podUID="3472bbbd-3579-4f37-a8f0-48fcbe1cb627" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:17:44.473695 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:44.473670 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b42983-f5cc57d49-hbnxp" Nov 03 00:17:44.546122 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:44.546050 5727 generic.go:356] "Generic (PLEG): container finished" podID="3472bbbd-3579-4f37-a8f0-48fcbe1cb627" containerID="3a5aacdce3f729603ead20ef22bf908a8fa99baf403cd81d70c2fbb9b1499b14" exitCode=0 Nov 03 00:17:44.546122 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:44.546105 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b42983-f5cc57d49-hbnxp" Nov 03 00:17:44.546342 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:44.546165 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b42983-f5cc57d49-hbnxp" event={"ID":"3472bbbd-3579-4f37-a8f0-48fcbe1cb627","Type":"ContainerDied","Data":"3a5aacdce3f729603ead20ef22bf908a8fa99baf403cd81d70c2fbb9b1499b14"} Nov 03 00:17:44.546342 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:44.546222 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b42983-f5cc57d49-hbnxp" event={"ID":"3472bbbd-3579-4f37-a8f0-48fcbe1cb627","Type":"ContainerDied","Data":"8ff7cd1e84e4ad9041ae85360c5d9420b3413f3ac4469a963968d3a0d6db7b41"} Nov 03 00:17:44.546342 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:44.546248 5727 scope.go:122] "RemoveContainer" containerID="3a5aacdce3f729603ead20ef22bf908a8fa99baf403cd81d70c2fbb9b1499b14" Nov 03 00:17:44.556889 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:44.556865 5727 scope.go:122] "RemoveContainer" containerID="3a5aacdce3f729603ead20ef22bf908a8fa99baf403cd81d70c2fbb9b1499b14" Nov 03 00:17:44.557196 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:17:44.557178 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a5aacdce3f729603ead20ef22bf908a8fa99baf403cd81d70c2fbb9b1499b14\": container with ID starting with 3a5aacdce3f729603ead20ef22bf908a8fa99baf403cd81d70c2fbb9b1499b14 not found: ID does not exist" containerID="3a5aacdce3f729603ead20ef22bf908a8fa99baf403cd81d70c2fbb9b1499b14" Nov 03 00:17:44.557274 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:44.557205 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a5aacdce3f729603ead20ef22bf908a8fa99baf403cd81d70c2fbb9b1499b14"} err="failed to get container status \"3a5aacdce3f729603ead20ef22bf908a8fa99baf403cd81d70c2fbb9b1499b14\": rpc error: code = NotFound desc = could not find container \"3a5aacdce3f729603ead20ef22bf908a8fa99baf403cd81d70c2fbb9b1499b14\": container with ID starting with 3a5aacdce3f729603ead20ef22bf908a8fa99baf403cd81d70c2fbb9b1499b14 not found: ID does not exist" Nov 03 00:17:44.579695 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:44.579669 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3472bbbd-3579-4f37-a8f0-48fcbe1cb627-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3472bbbd-3579-4f37-a8f0-48fcbe1cb627-plugins-config-volume\") pod \"3472bbbd-3579-4f37-a8f0-48fcbe1cb627\" (UID: \"3472bbbd-3579-4f37-a8f0-48fcbe1cb627\") " Nov 03 00:17:44.579806 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:44.579734 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3472bbbd-3579-4f37-a8f0-48fcbe1cb627-kube-api-access-wrrf7\" (UniqueName: \"kubernetes.io/projected/3472bbbd-3579-4f37-a8f0-48fcbe1cb627-kube-api-access-wrrf7\") pod \"3472bbbd-3579-4f37-a8f0-48fcbe1cb627\" (UID: \"3472bbbd-3579-4f37-a8f0-48fcbe1cb627\") " Nov 03 00:17:44.580028 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:44.580006 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3472bbbd-3579-4f37-a8f0-48fcbe1cb627-plugins-config-volume" pod "3472bbbd-3579-4f37-a8f0-48fcbe1cb627" (UID: "3472bbbd-3579-4f37-a8f0-48fcbe1cb627"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:17:44.582057 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:44.582036 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3472bbbd-3579-4f37-a8f0-48fcbe1cb627-kube-api-access-wrrf7" pod "3472bbbd-3579-4f37-a8f0-48fcbe1cb627" (UID: "3472bbbd-3579-4f37-a8f0-48fcbe1cb627"). InnerVolumeSpecName "kube-api-access-wrrf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:44.680573 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:44.680537 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3472bbbd-3579-4f37-a8f0-48fcbe1cb627-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:44.680573 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:44.680567 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrrf7\" (UniqueName: \"kubernetes.io/projected/3472bbbd-3579-4f37-a8f0-48fcbe1cb627-kube-api-access-wrrf7\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:44.967496 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:44.967452 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b42983-f5cc57d49-hbnxp"] Nov 03 00:17:44.972957 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:44.972927 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b42983-f5cc57d49-hbnxp"] Nov 03 00:17:45.882283 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:45.882248 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3472bbbd-3579-4f37-a8f0-48fcbe1cb627" path="/var/lib/kubelet/pods/3472bbbd-3579-4f37-a8f0-48fcbe1cb627/volumes" Nov 03 00:17:54.055473 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:54.055428 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-57f5d89f7c-zkm62_cf7f72a3-5ca3-4f5e-a8e2-c88f1485806a/authorino/0.log" Nov 03 00:17:54.406436 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:54.406401 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-ptd66_de8616c3-ba9c-4b99-813e-ef4c190fc4c7/manager/0.log" Nov 03 00:17:54.644647 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:54.644612 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-cp9j8_e5198e9e-a261-4491-93da-4fad7b359312/manager/0.log" Nov 03 00:17:54.875934 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:54.875908 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jr9qg_1394c83b-9059-4e3e-928f-bde172669b09/kuadrant-console-plugin/0.log" Nov 03 00:17:56.116149 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:56.116092 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-gxmgl_fd31caf7-718a-4852-9aeb-6dc5b8bd05c2/manager/0.log" Nov 03 00:17:56.558949 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:56.558866 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-x6h9v_349fa078-8c82-4c07-a596-900fbb745d7f/manager/1.log" Nov 03 00:17:57.238888 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:57.238773 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-x6h9v_349fa078-8c82-4c07-a596-900fbb745d7f/manager/0.log" Nov 03 00:17:57.824630 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:57.824592 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7fcfb78cb5-m2vnt_b2101b43-6e2c-42d8-8d60-d44faa6433f1/manager/0.log" Nov 03 00:17:57.846576 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:17:57.846543 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:17:58.072412 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:58.072381 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qprz9_b7266a2f-427b-47f5-9528-e8c111ae8d04/manager/0.log" Nov 03 00:17:58.303603 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:58.303523 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-knpxk_f601f493-313e-40b5-9e89-0c492a35255a/manager/0.log" Nov 03 00:17:59.692945 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:17:59.692920 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-cnzpg_b86548a3-cf54-41a9-8c65-f543ae9daa83/server/0.log" Nov 03 00:18:00.191903 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:00.191875 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-cb6tl_54f62e96-134f-44b3-839e-a02edd85ab08/proxy/0.log" Nov 03 00:18:15.599813 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.599774 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j"] Nov 03 00:18:15.600497 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.600477 5727 cpu_manager.go:414] "containerMap: removing container" podUID="3472bbbd-3579-4f37-a8f0-48fcbe1cb627" containerName="payload-processing" Nov 03 00:18:15.600557 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.600501 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3472bbbd-3579-4f37-a8f0-48fcbe1cb627" containerName="payload-processing" Nov 03 00:18:15.600557 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.600521 5727 cpu_manager.go:414] "containerMap: removing container" podUID="4d26cca8-ceef-40dd-a6ce-d60d2dd34c25" containerName="revoke-keys" Nov 03 00:18:15.600557 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.600528 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d26cca8-ceef-40dd-a6ce-d60d2dd34c25" containerName="revoke-keys" Nov 03 00:18:15.600557 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.600536 5727 cpu_manager.go:414] "containerMap: removing container" podUID="f83c8651-5db1-4e80-b9fb-ec5b793e7e94" containerName="main" Nov 03 00:18:15.600557 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.600542 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f83c8651-5db1-4e80-b9fb-ec5b793e7e94" containerName="main" Nov 03 00:18:15.600557 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.600553 5727 cpu_manager.go:414] "containerMap: removing container" podUID="49421404-0145-4fda-a9a4-26c7ba952374" containerName="payload-processing" Nov 03 00:18:15.600751 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.600558 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49421404-0145-4fda-a9a4-26c7ba952374" containerName="payload-processing" Nov 03 00:18:15.600751 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.600571 5727 cpu_manager.go:414] "containerMap: removing container" podUID="d6dcd3f9-d170-404b-9921-83b0d2e47ed4" containerName="payload-pre-processing" Nov 03 00:18:15.600751 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.600576 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6dcd3f9-d170-404b-9921-83b0d2e47ed4" containerName="payload-pre-processing" Nov 03 00:18:15.600751 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.600589 5727 cpu_manager.go:414] "containerMap: removing container" podUID="13206437-6c0e-4cbe-9f87-ce696dc65d6f" containerName="maas-api" Nov 03 00:18:15.600751 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.600593 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13206437-6c0e-4cbe-9f87-ce696dc65d6f" containerName="maas-api" Nov 03 00:18:15.600751 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.600601 5727 cpu_manager.go:414] "containerMap: removing container" podUID="d3799f21-d78b-468d-9347-ca3f0968409d" containerName="main" Nov 03 00:18:15.600751 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.600605 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3799f21-d78b-468d-9347-ca3f0968409d" containerName="main" Nov 03 00:18:15.600751 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.600615 5727 cpu_manager.go:414] "containerMap: removing container" podUID="940cf7dc-acaa-4ff3-b473-99c7536913cf" containerName="payload-processing" Nov 03 00:18:15.600751 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.600619 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="940cf7dc-acaa-4ff3-b473-99c7536913cf" containerName="payload-processing" Nov 03 00:18:15.600751 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.600627 5727 cpu_manager.go:414] "containerMap: removing container" podUID="4bd218d6-1000-4fce-bade-c54fa8e7026b" containerName="payload-pre-processing" Nov 03 00:18:15.600751 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.600632 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bd218d6-1000-4fce-bade-c54fa8e7026b" containerName="payload-pre-processing" Nov 03 00:18:15.600751 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.600707 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="940cf7dc-acaa-4ff3-b473-99c7536913cf" containerName="payload-processing" Nov 03 00:18:15.600751 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.600715 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bd218d6-1000-4fce-bade-c54fa8e7026b" containerName="payload-pre-processing" Nov 03 00:18:15.600751 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.600724 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="3472bbbd-3579-4f37-a8f0-48fcbe1cb627" containerName="payload-processing" Nov 03 00:18:15.600751 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.600730 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d26cca8-ceef-40dd-a6ce-d60d2dd34c25" containerName="revoke-keys" Nov 03 00:18:15.600751 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.600736 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="f83c8651-5db1-4e80-b9fb-ec5b793e7e94" containerName="main" Nov 03 00:18:15.600751 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.600741 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="49421404-0145-4fda-a9a4-26c7ba952374" containerName="payload-processing" Nov 03 00:18:15.600751 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.600748 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6dcd3f9-d170-404b-9921-83b0d2e47ed4" containerName="payload-pre-processing" Nov 03 00:18:15.600751 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.600755 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="13206437-6c0e-4cbe-9f87-ce696dc65d6f" containerName="maas-api" Nov 03 00:18:15.601337 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.600761 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3799f21-d78b-468d-9347-ca3f0968409d" containerName="main" Nov 03 00:18:15.603412 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.603396 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j" Nov 03 00:18:15.660840 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.660800 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b5e4655-f1d4-4369-85f2-77959919447e-maas-api-tls\") pod \"maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j\" (UID: \"1b5e4655-f1d4-4369-85f2-77959919447e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j" Nov 03 00:18:15.661047 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.660851 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsq7p\" (UniqueName: \"kubernetes.io/projected/1b5e4655-f1d4-4369-85f2-77959919447e-kube-api-access-xsq7p\") pod \"maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j\" (UID: \"1b5e4655-f1d4-4369-85f2-77959919447e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j" Nov 03 00:18:15.661047 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.660969 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b5e4655-f1d4-4369-85f2-77959919447e-openshift-service-ca\") pod \"maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j\" (UID: \"1b5e4655-f1d4-4369-85f2-77959919447e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j" Nov 03 00:18:15.661189 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.661066 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b5e4655-f1d4-4369-85f2-77959919447e-metrics-certs\") pod \"maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j\" (UID: \"1b5e4655-f1d4-4369-85f2-77959919447e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j" Nov 03 00:18:15.704777 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.704732 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-018f8381-serving-cert\"" Nov 03 00:18:15.704979 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.704911 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:18:15.704979 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.704931 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:18:15.704979 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.704926 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wm4gz\"" Nov 03 00:18:15.705264 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.705006 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 00:18:15.707765 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.707737 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j"] Nov 03 00:18:15.762115 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.762072 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b5e4655-f1d4-4369-85f2-77959919447e-maas-api-tls\") pod \"maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j\" (UID: \"1b5e4655-f1d4-4369-85f2-77959919447e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j" Nov 03 00:18:15.762489 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.762459 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsq7p\" (UniqueName: \"kubernetes.io/projected/1b5e4655-f1d4-4369-85f2-77959919447e-kube-api-access-xsq7p\") pod \"maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j\" (UID: \"1b5e4655-f1d4-4369-85f2-77959919447e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j" Nov 03 00:18:15.762701 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.762680 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b5e4655-f1d4-4369-85f2-77959919447e-openshift-service-ca\") pod \"maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j\" (UID: \"1b5e4655-f1d4-4369-85f2-77959919447e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j" Nov 03 00:18:15.762923 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.762905 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b5e4655-f1d4-4369-85f2-77959919447e-metrics-certs\") pod \"maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j\" (UID: \"1b5e4655-f1d4-4369-85f2-77959919447e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j" Nov 03 00:18:15.766473 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:18:15.766443 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-018f8381-serving-cert: secret "maas-api-e2e-ait-018f8381-serving-cert" not found Nov 03 00:18:15.766791 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:18:15.766768 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b5e4655-f1d4-4369-85f2-77959919447e-maas-api-tls podName:1b5e4655-f1d4-4369-85f2-77959919447e nodeName:}" failed. No retries permitted until 2026-11-03 00:18:16.266741966 +0000 UTC m=+5382.505239618 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1b5e4655-f1d4-4369-85f2-77959919447e-maas-api-tls") pod "maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j" (UID: "1b5e4655-f1d4-4369-85f2-77959919447e") : secret "maas-api-e2e-ait-018f8381-serving-cert" not found Nov 03 00:18:15.767214 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.767109 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b5e4655-f1d4-4369-85f2-77959919447e-openshift-service-ca\") pod \"maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j\" (UID: \"1b5e4655-f1d4-4369-85f2-77959919447e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j" Nov 03 00:18:15.767309 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.767264 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b5e4655-f1d4-4369-85f2-77959919447e-metrics-certs\") pod \"maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j\" (UID: \"1b5e4655-f1d4-4369-85f2-77959919447e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j" Nov 03 00:18:15.829628 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:15.829586 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsq7p\" (UniqueName: \"kubernetes.io/projected/1b5e4655-f1d4-4369-85f2-77959919447e-kube-api-access-xsq7p\") pod \"maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j\" (UID: \"1b5e4655-f1d4-4369-85f2-77959919447e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j" Nov 03 00:18:16.367664 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:16.367626 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b5e4655-f1d4-4369-85f2-77959919447e-maas-api-tls\") pod \"maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j\" (UID: \"1b5e4655-f1d4-4369-85f2-77959919447e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j" Nov 03 00:18:16.370289 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:16.370260 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b5e4655-f1d4-4369-85f2-77959919447e-maas-api-tls\") pod \"maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j\" (UID: \"1b5e4655-f1d4-4369-85f2-77959919447e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j" Nov 03 00:18:16.517311 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:16.517278 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j" Nov 03 00:18:16.894338 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:16.894310 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j"] Nov 03 00:18:16.896344 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:18:16.896316 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b5e4655_f1d4_4369_85f2_77959919447e.slice/crio-3e293484745267722feb8c46dd35db2158d12fe934e77bcb2974b06a34f88df8 WatchSource:0}: Error finding container 3e293484745267722feb8c46dd35db2158d12fe934e77bcb2974b06a34f88df8: Status 404 returned error can't find the container with id 3e293484745267722feb8c46dd35db2158d12fe934e77bcb2974b06a34f88df8 Nov 03 00:18:16.897903 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:16.897882 5727 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:18:17.769961 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:17.769922 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j" event={"ID":"1b5e4655-f1d4-4369-85f2-77959919447e","Type":"ContainerStarted","Data":"3e293484745267722feb8c46dd35db2158d12fe934e77bcb2974b06a34f88df8"} Nov 03 00:18:18.777499 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:18.777460 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j" event={"ID":"1b5e4655-f1d4-4369-85f2-77959919447e","Type":"ContainerStarted","Data":"33dfe132cd9004bcc64aed40216b316782aba3c44e28f1060adcf998d396d2be"} Nov 03 00:18:18.777913 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:18.777663 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j" Nov 03 00:18:18.896823 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:18.896768 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j" podStartSLOduration=2.7767667019999998 podStartE2EDuration="3.896754679s" totalImagesPullingTime="1.119987977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:18:18.89468552 +0000 UTC m=+5385.133183183" watchObservedRunningTime="2026-11-03 00:18:18.896754679 +0000 UTC m=+5385.135252342" Nov 03 00:18:24.789322 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:24.789286 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j" Nov 03 00:18:27.845846 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:18:27.845810 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:18:35.064748 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:35.064716 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 03 00:18:35.080845 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:35.080814 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 03 00:18:35.178688 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:35.178660 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 03 00:18:35.194710 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:18:35.194680 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 03 00:18:57.846060 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:18:57.846011 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:19:05.468458 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:19:05.468420 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j"] Nov 03 00:19:05.469044 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:19:05.468612 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j" podUID="1b5e4655-f1d4-4369-85f2-77959919447e" containerName="maas-api" containerID="cri-o://33dfe132cd9004bcc64aed40216b316782aba3c44e28f1060adcf998d396d2be" gracePeriod=30 Nov 03 00:19:05.731575 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:19:05.731520 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j" Nov 03 00:19:05.837532 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:19:05.837495 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b5e4655-f1d4-4369-85f2-77959919447e-kube-api-access-xsq7p\" (UniqueName: \"kubernetes.io/projected/1b5e4655-f1d4-4369-85f2-77959919447e-kube-api-access-xsq7p\") pod \"1b5e4655-f1d4-4369-85f2-77959919447e\" (UID: \"1b5e4655-f1d4-4369-85f2-77959919447e\") " Nov 03 00:19:05.838533 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:19:05.837551 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b5e4655-f1d4-4369-85f2-77959919447e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b5e4655-f1d4-4369-85f2-77959919447e-maas-api-tls\") pod \"1b5e4655-f1d4-4369-85f2-77959919447e\" (UID: \"1b5e4655-f1d4-4369-85f2-77959919447e\") " Nov 03 00:19:05.838856 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:19:05.838826 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b5e4655-f1d4-4369-85f2-77959919447e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b5e4655-f1d4-4369-85f2-77959919447e-openshift-service-ca\") pod \"1b5e4655-f1d4-4369-85f2-77959919447e\" (UID: \"1b5e4655-f1d4-4369-85f2-77959919447e\") " Nov 03 00:19:05.839182 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:19:05.839162 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b5e4655-f1d4-4369-85f2-77959919447e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b5e4655-f1d4-4369-85f2-77959919447e-metrics-certs\") pod \"1b5e4655-f1d4-4369-85f2-77959919447e\" (UID: \"1b5e4655-f1d4-4369-85f2-77959919447e\") " Nov 03 00:19:05.839761 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:19:05.839679 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b5e4655-f1d4-4369-85f2-77959919447e-openshift-service-ca" pod "1b5e4655-f1d4-4369-85f2-77959919447e" (UID: "1b5e4655-f1d4-4369-85f2-77959919447e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:19:05.841769 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:19:05.841724 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b5e4655-f1d4-4369-85f2-77959919447e-maas-api-tls" pod "1b5e4655-f1d4-4369-85f2-77959919447e" (UID: "1b5e4655-f1d4-4369-85f2-77959919447e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:19:05.841999 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:19:05.841966 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b5e4655-f1d4-4369-85f2-77959919447e-kube-api-access-xsq7p" pod "1b5e4655-f1d4-4369-85f2-77959919447e" (UID: "1b5e4655-f1d4-4369-85f2-77959919447e"). InnerVolumeSpecName "kube-api-access-xsq7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:19:05.842090 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:19:05.842045 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b5e4655-f1d4-4369-85f2-77959919447e-metrics-certs" pod "1b5e4655-f1d4-4369-85f2-77959919447e" (UID: "1b5e4655-f1d4-4369-85f2-77959919447e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:19:05.940734 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:19:05.940704 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsq7p\" (UniqueName: \"kubernetes.io/projected/1b5e4655-f1d4-4369-85f2-77959919447e-kube-api-access-xsq7p\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:05.940734 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:19:05.940733 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b5e4655-f1d4-4369-85f2-77959919447e-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:05.940920 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:19:05.940744 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b5e4655-f1d4-4369-85f2-77959919447e-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:05.940920 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:19:05.940755 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b5e4655-f1d4-4369-85f2-77959919447e-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:06.071185 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:19:06.071152 5727 generic.go:356] "Generic (PLEG): container finished" podID="1b5e4655-f1d4-4369-85f2-77959919447e" containerID="33dfe132cd9004bcc64aed40216b316782aba3c44e28f1060adcf998d396d2be" exitCode=0 Nov 03 00:19:06.071375 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:19:06.071205 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j" Nov 03 00:19:06.071375 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:19:06.071242 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j" event={"ID":"1b5e4655-f1d4-4369-85f2-77959919447e","Type":"ContainerDied","Data":"33dfe132cd9004bcc64aed40216b316782aba3c44e28f1060adcf998d396d2be"} Nov 03 00:19:06.071375 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:19:06.071300 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j" event={"ID":"1b5e4655-f1d4-4369-85f2-77959919447e","Type":"ContainerDied","Data":"3e293484745267722feb8c46dd35db2158d12fe934e77bcb2974b06a34f88df8"} Nov 03 00:19:06.071375 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:19:06.071329 5727 scope.go:122] "RemoveContainer" containerID="33dfe132cd9004bcc64aed40216b316782aba3c44e28f1060adcf998d396d2be" Nov 03 00:19:06.081164 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:19:06.081148 5727 scope.go:122] "RemoveContainer" containerID="33dfe132cd9004bcc64aed40216b316782aba3c44e28f1060adcf998d396d2be" Nov 03 00:19:06.081449 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:19:06.081426 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33dfe132cd9004bcc64aed40216b316782aba3c44e28f1060adcf998d396d2be\": container with ID starting with 33dfe132cd9004bcc64aed40216b316782aba3c44e28f1060adcf998d396d2be not found: ID does not exist" containerID="33dfe132cd9004bcc64aed40216b316782aba3c44e28f1060adcf998d396d2be" Nov 03 00:19:06.081501 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:19:06.081459 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33dfe132cd9004bcc64aed40216b316782aba3c44e28f1060adcf998d396d2be"} err="failed to get container status \"33dfe132cd9004bcc64aed40216b316782aba3c44e28f1060adcf998d396d2be\": rpc error: code = NotFound desc = could not find container \"33dfe132cd9004bcc64aed40216b316782aba3c44e28f1060adcf998d396d2be\": container with ID starting with 33dfe132cd9004bcc64aed40216b316782aba3c44e28f1060adcf998d396d2be not found: ID does not exist" Nov 03 00:19:06.237272 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:19:06.237240 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j"] Nov 03 00:19:06.240992 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:19:06.240968 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-018f8381-68c94fbdb6-cvc4j"] Nov 03 00:19:07.887814 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:19:07.887773 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b5e4655-f1d4-4369-85f2-77959919447e" path="/var/lib/kubelet/pods/1b5e4655-f1d4-4369-85f2-77959919447e/volumes" Nov 03 00:19:27.845420 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:19:27.845376 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:19:57.845091 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:19:57.845053 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:20:26.923974 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:26.923931 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jh6w4"] Nov 03 00:20:26.924634 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:26.924608 5727 cpu_manager.go:414] "containerMap: removing container" podUID="1b5e4655-f1d4-4369-85f2-77959919447e" containerName="maas-api" Nov 03 00:20:26.924716 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:26.924637 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b5e4655-f1d4-4369-85f2-77959919447e" containerName="maas-api" Nov 03 00:20:26.924809 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:26.924796 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b5e4655-f1d4-4369-85f2-77959919447e" containerName="maas-api" Nov 03 00:20:26.927942 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:26.927921 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jh6w4" Nov 03 00:20:26.977287 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:26.977260 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dda39dc-5fba-4455-b44a-4e2e248668ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-jh6w4\" (UID: \"9dda39dc-5fba-4455-b44a-4e2e248668ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jh6w4" Nov 03 00:20:26.977391 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:26.977305 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dda39dc-5fba-4455-b44a-4e2e248668ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-jh6w4\" (UID: \"9dda39dc-5fba-4455-b44a-4e2e248668ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jh6w4" Nov 03 00:20:26.977447 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:26.977425 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsqhn\" (UniqueName: \"kubernetes.io/projected/9dda39dc-5fba-4455-b44a-4e2e248668ac-kube-api-access-qsqhn\") pod \"maas-api-revoke-sub-f904bff9-jh6w4\" (UID: \"9dda39dc-5fba-4455-b44a-4e2e248668ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jh6w4" Nov 03 00:20:26.980285 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:26.980258 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:20:27.029756 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:27.029729 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:20:27.029756 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:27.029729 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:20:27.030011 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:27.029771 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:20:27.032941 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:27.032920 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jh6w4"] Nov 03 00:20:27.078600 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:27.078574 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dda39dc-5fba-4455-b44a-4e2e248668ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-jh6w4\" (UID: \"9dda39dc-5fba-4455-b44a-4e2e248668ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jh6w4" Nov 03 00:20:27.078738 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:27.078615 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dda39dc-5fba-4455-b44a-4e2e248668ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-jh6w4\" (UID: \"9dda39dc-5fba-4455-b44a-4e2e248668ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jh6w4" Nov 03 00:20:27.078738 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:27.078656 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsqhn\" (UniqueName: \"kubernetes.io/projected/9dda39dc-5fba-4455-b44a-4e2e248668ac-kube-api-access-qsqhn\") pod \"maas-api-revoke-sub-f904bff9-jh6w4\" (UID: \"9dda39dc-5fba-4455-b44a-4e2e248668ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jh6w4" Nov 03 00:20:27.079250 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:27.079225 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dda39dc-5fba-4455-b44a-4e2e248668ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-jh6w4\" (UID: \"9dda39dc-5fba-4455-b44a-4e2e248668ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jh6w4" Nov 03 00:20:27.081256 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:27.081229 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dda39dc-5fba-4455-b44a-4e2e248668ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-jh6w4\" (UID: \"9dda39dc-5fba-4455-b44a-4e2e248668ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jh6w4" Nov 03 00:20:27.139275 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:27.139248 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsqhn\" (UniqueName: \"kubernetes.io/projected/9dda39dc-5fba-4455-b44a-4e2e248668ac-kube-api-access-qsqhn\") pod \"maas-api-revoke-sub-f904bff9-jh6w4\" (UID: \"9dda39dc-5fba-4455-b44a-4e2e248668ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jh6w4" Nov 03 00:20:27.241498 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:27.241416 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jh6w4" Nov 03 00:20:27.618707 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:27.618680 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jh6w4"] Nov 03 00:20:27.620684 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:20:27.620655 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dda39dc_5fba_4455_b44a_4e2e248668ac.slice/crio-8bb894ec996d6f93bfbcb14002626ebc6af05ea5d9a137e0e8f95c1578a3ba3c WatchSource:0}: Error finding container 8bb894ec996d6f93bfbcb14002626ebc6af05ea5d9a137e0e8f95c1578a3ba3c: Status 404 returned error can't find the container with id 8bb894ec996d6f93bfbcb14002626ebc6af05ea5d9a137e0e8f95c1578a3ba3c Nov 03 00:20:27.846269 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:20:27.846233 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:20:28.578312 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:28.578281 5727 generic.go:356] "Generic (PLEG): container finished" podID="9dda39dc-5fba-4455-b44a-4e2e248668ac" containerID="5ee034032f0a67fcd154e8bf670bc0eb4d18e966d0a01031901c19dc118672f0" exitCode=0 Nov 03 00:20:28.578674 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:28.578369 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jh6w4" event={"ID":"9dda39dc-5fba-4455-b44a-4e2e248668ac","Type":"ContainerDied","Data":"5ee034032f0a67fcd154e8bf670bc0eb4d18e966d0a01031901c19dc118672f0"} Nov 03 00:20:28.578674 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:28.578428 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jh6w4" event={"ID":"9dda39dc-5fba-4455-b44a-4e2e248668ac","Type":"ContainerStarted","Data":"8bb894ec996d6f93bfbcb14002626ebc6af05ea5d9a137e0e8f95c1578a3ba3c"} Nov 03 00:20:29.056387 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:29.056351 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8c79816-5kg5f"] Nov 03 00:20:29.059344 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:29.059318 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8c79816-5kg5f"] Nov 03 00:20:29.726656 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:29.726630 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jh6w4" Nov 03 00:20:29.808115 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:29.808085 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dda39dc-5fba-4455-b44a-4e2e248668ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dda39dc-5fba-4455-b44a-4e2e248668ac-maas-api-openshift-ca\") pod \"9dda39dc-5fba-4455-b44a-4e2e248668ac\" (UID: \"9dda39dc-5fba-4455-b44a-4e2e248668ac\") " Nov 03 00:20:29.808287 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:29.808162 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dda39dc-5fba-4455-b44a-4e2e248668ac-kube-api-access-qsqhn\" (UniqueName: \"kubernetes.io/projected/9dda39dc-5fba-4455-b44a-4e2e248668ac-kube-api-access-qsqhn\") pod \"9dda39dc-5fba-4455-b44a-4e2e248668ac\" (UID: \"9dda39dc-5fba-4455-b44a-4e2e248668ac\") " Nov 03 00:20:29.808287 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:29.808264 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dda39dc-5fba-4455-b44a-4e2e248668ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dda39dc-5fba-4455-b44a-4e2e248668ac-maas-api-xks-ca\") pod \"9dda39dc-5fba-4455-b44a-4e2e248668ac\" (UID: \"9dda39dc-5fba-4455-b44a-4e2e248668ac\") " Nov 03 00:20:29.808458 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:29.808433 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dda39dc-5fba-4455-b44a-4e2e248668ac-maas-api-openshift-ca" pod "9dda39dc-5fba-4455-b44a-4e2e248668ac" (UID: "9dda39dc-5fba-4455-b44a-4e2e248668ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:20:29.808555 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:29.808540 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dda39dc-5fba-4455-b44a-4e2e248668ac-maas-api-openshift-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:20:29.810390 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:29.810370 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dda39dc-5fba-4455-b44a-4e2e248668ac-kube-api-access-qsqhn" pod "9dda39dc-5fba-4455-b44a-4e2e248668ac" (UID: "9dda39dc-5fba-4455-b44a-4e2e248668ac"). InnerVolumeSpecName "kube-api-access-qsqhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:20:29.810468 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:29.810403 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dda39dc-5fba-4455-b44a-4e2e248668ac-maas-api-xks-ca" pod "9dda39dc-5fba-4455-b44a-4e2e248668ac" (UID: "9dda39dc-5fba-4455-b44a-4e2e248668ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:20:29.882528 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:29.882452 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="990be195-b421-4192-8136-d216d3b22e06" path="/var/lib/kubelet/pods/990be195-b421-4192-8136-d216d3b22e06/volumes" Nov 03 00:20:29.909366 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:29.909338 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dda39dc-5fba-4455-b44a-4e2e248668ac-maas-api-xks-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:20:29.909366 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:29.909365 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsqhn\" (UniqueName: \"kubernetes.io/projected/9dda39dc-5fba-4455-b44a-4e2e248668ac-kube-api-access-qsqhn\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:20:30.590708 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:30.590673 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jh6w4" event={"ID":"9dda39dc-5fba-4455-b44a-4e2e248668ac","Type":"ContainerDied","Data":"8bb894ec996d6f93bfbcb14002626ebc6af05ea5d9a137e0e8f95c1578a3ba3c"} Nov 03 00:20:30.590914 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:30.590728 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bb894ec996d6f93bfbcb14002626ebc6af05ea5d9a137e0e8f95c1578a3ba3c" Nov 03 00:20:30.590914 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:30.590689 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jh6w4" Nov 03 00:20:41.051200 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.051159 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ee97e86-tqnrv"] Nov 03 00:20:41.100601 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.100563 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ee97e86-tqnrv"] Nov 03 00:20:41.102702 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.102677 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2"] Nov 03 00:20:41.103150 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.103121 5727 cpu_manager.go:414] "containerMap: removing container" podUID="9dda39dc-5fba-4455-b44a-4e2e248668ac" containerName="revoke-keys" Nov 03 00:20:41.103200 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.103152 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dda39dc-5fba-4455-b44a-4e2e248668ac" containerName="revoke-keys" Nov 03 00:20:41.103246 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.103242 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dda39dc-5fba-4455-b44a-4e2e248668ac" containerName="revoke-keys" Nov 03 00:20:41.107550 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.107532 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:41.203465 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.203419 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2"] Nov 03 00:20:41.209222 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.209190 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3137ed7b-3a02-433a-b95d-7cd75469aac5-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-vgtv2\" (UID: \"3137ed7b-3a02-433a-b95d-7cd75469aac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:41.209777 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.209236 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqmlf\" (UniqueName: \"kubernetes.io/projected/3137ed7b-3a02-433a-b95d-7cd75469aac5-kube-api-access-tqmlf\") pod \"maas-default-gateway-openshift-default-694db55c87-vgtv2\" (UID: \"3137ed7b-3a02-433a-b95d-7cd75469aac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:41.209777 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.209333 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3137ed7b-3a02-433a-b95d-7cd75469aac5-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-vgtv2\" (UID: \"3137ed7b-3a02-433a-b95d-7cd75469aac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:41.209777 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.209391 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3137ed7b-3a02-433a-b95d-7cd75469aac5-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-vgtv2\" (UID: \"3137ed7b-3a02-433a-b95d-7cd75469aac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:41.209777 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.209441 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3137ed7b-3a02-433a-b95d-7cd75469aac5-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-vgtv2\" (UID: \"3137ed7b-3a02-433a-b95d-7cd75469aac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:41.209777 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.209528 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3137ed7b-3a02-433a-b95d-7cd75469aac5-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-vgtv2\" (UID: \"3137ed7b-3a02-433a-b95d-7cd75469aac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:41.209777 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.209568 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3137ed7b-3a02-433a-b95d-7cd75469aac5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-vgtv2\" (UID: \"3137ed7b-3a02-433a-b95d-7cd75469aac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:41.209777 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.209595 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3137ed7b-3a02-433a-b95d-7cd75469aac5-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-vgtv2\" (UID: \"3137ed7b-3a02-433a-b95d-7cd75469aac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:41.209777 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.209630 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3137ed7b-3a02-433a-b95d-7cd75469aac5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-vgtv2\" (UID: \"3137ed7b-3a02-433a-b95d-7cd75469aac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:41.310812 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.310717 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3137ed7b-3a02-433a-b95d-7cd75469aac5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-vgtv2\" (UID: \"3137ed7b-3a02-433a-b95d-7cd75469aac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:41.310812 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.310770 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3137ed7b-3a02-433a-b95d-7cd75469aac5-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-vgtv2\" (UID: \"3137ed7b-3a02-433a-b95d-7cd75469aac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:41.310812 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.310794 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3137ed7b-3a02-433a-b95d-7cd75469aac5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-vgtv2\" (UID: \"3137ed7b-3a02-433a-b95d-7cd75469aac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:41.311083 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.310888 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3137ed7b-3a02-433a-b95d-7cd75469aac5-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-vgtv2\" (UID: \"3137ed7b-3a02-433a-b95d-7cd75469aac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:41.311083 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.310919 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqmlf\" (UniqueName: \"kubernetes.io/projected/3137ed7b-3a02-433a-b95d-7cd75469aac5-kube-api-access-tqmlf\") pod \"maas-default-gateway-openshift-default-694db55c87-vgtv2\" (UID: \"3137ed7b-3a02-433a-b95d-7cd75469aac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:41.311083 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.310945 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3137ed7b-3a02-433a-b95d-7cd75469aac5-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-vgtv2\" (UID: \"3137ed7b-3a02-433a-b95d-7cd75469aac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:41.311083 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.310961 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3137ed7b-3a02-433a-b95d-7cd75469aac5-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-vgtv2\" (UID: \"3137ed7b-3a02-433a-b95d-7cd75469aac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:41.311083 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.310996 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3137ed7b-3a02-433a-b95d-7cd75469aac5-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-vgtv2\" (UID: \"3137ed7b-3a02-433a-b95d-7cd75469aac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:41.311083 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.311025 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3137ed7b-3a02-433a-b95d-7cd75469aac5-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-vgtv2\" (UID: \"3137ed7b-3a02-433a-b95d-7cd75469aac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:41.311481 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.311347 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3137ed7b-3a02-433a-b95d-7cd75469aac5-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-vgtv2\" (UID: \"3137ed7b-3a02-433a-b95d-7cd75469aac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:41.311481 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.311432 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3137ed7b-3a02-433a-b95d-7cd75469aac5-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-vgtv2\" (UID: \"3137ed7b-3a02-433a-b95d-7cd75469aac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:41.311602 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.311556 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3137ed7b-3a02-433a-b95d-7cd75469aac5-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-vgtv2\" (UID: \"3137ed7b-3a02-433a-b95d-7cd75469aac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:41.311670 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.311624 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3137ed7b-3a02-433a-b95d-7cd75469aac5-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-vgtv2\" (UID: \"3137ed7b-3a02-433a-b95d-7cd75469aac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:41.312026 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.311999 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3137ed7b-3a02-433a-b95d-7cd75469aac5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-vgtv2\" (UID: \"3137ed7b-3a02-433a-b95d-7cd75469aac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:41.313380 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.313357 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3137ed7b-3a02-433a-b95d-7cd75469aac5-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-vgtv2\" (UID: \"3137ed7b-3a02-433a-b95d-7cd75469aac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:41.313625 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.313608 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3137ed7b-3a02-433a-b95d-7cd75469aac5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-vgtv2\" (UID: \"3137ed7b-3a02-433a-b95d-7cd75469aac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:41.371901 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.371867 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3137ed7b-3a02-433a-b95d-7cd75469aac5-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-vgtv2\" (UID: \"3137ed7b-3a02-433a-b95d-7cd75469aac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:41.372098 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.371973 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqmlf\" (UniqueName: \"kubernetes.io/projected/3137ed7b-3a02-433a-b95d-7cd75469aac5-kube-api-access-tqmlf\") pod \"maas-default-gateway-openshift-default-694db55c87-vgtv2\" (UID: \"3137ed7b-3a02-433a-b95d-7cd75469aac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:41.421435 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.421395 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:41.849967 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.849935 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2"] Nov 03 00:20:41.851894 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:20:41.851862 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3137ed7b_3a02_433a_b95d_7cd75469aac5.slice/crio-642badc71604ff68fbdb35a361f6f263876da1e7c94b67d1ac3c109a46452a7f WatchSource:0}: Error finding container 642badc71604ff68fbdb35a361f6f263876da1e7c94b67d1ac3c109a46452a7f: Status 404 returned error can't find the container with id 642badc71604ff68fbdb35a361f6f263876da1e7c94b67d1ac3c109a46452a7f Nov 03 00:20:41.854062 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.854031 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 00:20:41.854121 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.854100 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 00:20:41.854191 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.854152 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 00:20:41.889590 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:41.888426 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dd412c7-c93e-4cbc-bf1d-2708537a8c25" path="/var/lib/kubelet/pods/4dd412c7-c93e-4cbc-bf1d-2708537a8c25/volumes" Nov 03 00:20:42.682114 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:42.682065 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" event={"ID":"3137ed7b-3a02-433a-b95d-7cd75469aac5","Type":"ContainerStarted","Data":"d1e70d4165596d7f1678aee40fabaa4eb8af75a242d554d1051e01805b60929e"} Nov 03 00:20:42.682577 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:42.682168 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" event={"ID":"3137ed7b-3a02-433a-b95d-7cd75469aac5","Type":"ContainerStarted","Data":"642badc71604ff68fbdb35a361f6f263876da1e7c94b67d1ac3c109a46452a7f"} Nov 03 00:20:42.871156 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:42.854368 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" podStartSLOduration=1.8543493149999999 podStartE2EDuration="1.854349315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:42.851709499 +0000 UTC m=+5529.090207182" watchObservedRunningTime="2026-11-03 00:20:42.854349315 +0000 UTC m=+5529.092846979" Nov 03 00:20:43.422648 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:43.422601 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:43.424103 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:43.424075 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-vgtv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 03 00:20:43.424252 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:43.424147 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" podUID="3137ed7b-3a02-433a-b95d-7cd75469aac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 03 00:20:44.422382 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:44.422341 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-vgtv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 03 00:20:44.422863 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:44.422400 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" podUID="3137ed7b-3a02-433a-b95d-7cd75469aac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 03 00:20:45.422422 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:45.422385 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-vgtv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 03 00:20:45.422952 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:45.422448 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" podUID="3137ed7b-3a02-433a-b95d-7cd75469aac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 03 00:20:46.422205 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:46.422160 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-vgtv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 03 00:20:46.422410 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:46.422222 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" podUID="3137ed7b-3a02-433a-b95d-7cd75469aac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 03 00:20:47.422700 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:47.422658 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-vgtv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 03 00:20:47.423261 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:47.422723 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" podUID="3137ed7b-3a02-433a-b95d-7cd75469aac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 03 00:20:48.422427 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:48.422395 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-vgtv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 03 00:20:48.422604 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:48.422451 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" podUID="3137ed7b-3a02-433a-b95d-7cd75469aac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 03 00:20:49.422799 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:49.422764 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-vgtv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 03 00:20:49.423231 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:49.422820 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" podUID="3137ed7b-3a02-433a-b95d-7cd75469aac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 03 00:20:51.462303 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:51.462259 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-vgtv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:20:51.462831 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:51.462334 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" podUID="3137ed7b-3a02-433a-b95d-7cd75469aac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:20:52.045993 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:52.045963 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d31257c-2d46d"] Nov 03 00:20:52.096653 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:52.096616 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d31257c-2d46d"] Nov 03 00:20:52.414888 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:52.414846 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:52.415202 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:52.415171 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:52.415933 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:52.415915 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vgtv2" Nov 03 00:20:53.884596 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:53.884561 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d26cca8-ceef-40dd-a6ce-d60d2dd34c25" path="/var/lib/kubelet/pods/4d26cca8-ceef-40dd-a6ce-d60d2dd34c25/volumes" Nov 03 00:20:56.164865 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.164826 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4"] Nov 03 00:20:56.170691 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.170666 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:20:56.207386 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.207339 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5514e1f-1fb4-4158-85c8-f17fe2b94eb8-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-kvws4\" (UID: \"a5514e1f-1fb4-4158-85c8-f17fe2b94eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:20:56.207719 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.207662 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5514e1f-1fb4-4158-85c8-f17fe2b94eb8-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-kvws4\" (UID: \"a5514e1f-1fb4-4158-85c8-f17fe2b94eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:20:56.207990 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.207904 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5514e1f-1fb4-4158-85c8-f17fe2b94eb8-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-kvws4\" (UID: \"a5514e1f-1fb4-4158-85c8-f17fe2b94eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:20:56.208439 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.208153 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5514e1f-1fb4-4158-85c8-f17fe2b94eb8-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-kvws4\" (UID: \"a5514e1f-1fb4-4158-85c8-f17fe2b94eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:20:56.208439 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.208207 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5514e1f-1fb4-4158-85c8-f17fe2b94eb8-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-kvws4\" (UID: \"a5514e1f-1fb4-4158-85c8-f17fe2b94eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:20:56.208439 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.208236 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-477x5\" (UniqueName: \"kubernetes.io/projected/a5514e1f-1fb4-4158-85c8-f17fe2b94eb8-kube-api-access-477x5\") pod \"maas-default-gateway-openshift-default-694db55c87-kvws4\" (UID: \"a5514e1f-1fb4-4158-85c8-f17fe2b94eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:20:56.208439 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.208298 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5514e1f-1fb4-4158-85c8-f17fe2b94eb8-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-kvws4\" (UID: \"a5514e1f-1fb4-4158-85c8-f17fe2b94eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:20:56.208439 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.208373 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5514e1f-1fb4-4158-85c8-f17fe2b94eb8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-kvws4\" (UID: \"a5514e1f-1fb4-4158-85c8-f17fe2b94eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:20:56.208986 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.208401 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5514e1f-1fb4-4158-85c8-f17fe2b94eb8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-kvws4\" (UID: \"a5514e1f-1fb4-4158-85c8-f17fe2b94eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:20:56.279717 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.279682 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4"] Nov 03 00:20:56.312752 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.312720 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5514e1f-1fb4-4158-85c8-f17fe2b94eb8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-kvws4\" (UID: \"a5514e1f-1fb4-4158-85c8-f17fe2b94eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:20:56.313030 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.312994 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5514e1f-1fb4-4158-85c8-f17fe2b94eb8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-kvws4\" (UID: \"a5514e1f-1fb4-4158-85c8-f17fe2b94eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:20:56.313270 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.313251 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5514e1f-1fb4-4158-85c8-f17fe2b94eb8-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-kvws4\" (UID: \"a5514e1f-1fb4-4158-85c8-f17fe2b94eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:20:56.313434 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.313416 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5514e1f-1fb4-4158-85c8-f17fe2b94eb8-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-kvws4\" (UID: \"a5514e1f-1fb4-4158-85c8-f17fe2b94eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:20:56.313620 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.313593 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5514e1f-1fb4-4158-85c8-f17fe2b94eb8-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-kvws4\" (UID: \"a5514e1f-1fb4-4158-85c8-f17fe2b94eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:20:56.313768 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.313749 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5514e1f-1fb4-4158-85c8-f17fe2b94eb8-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-kvws4\" (UID: \"a5514e1f-1fb4-4158-85c8-f17fe2b94eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:20:56.317146 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.313911 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5514e1f-1fb4-4158-85c8-f17fe2b94eb8-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-kvws4\" (UID: \"a5514e1f-1fb4-4158-85c8-f17fe2b94eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:20:56.321637 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.320714 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-477x5\" (UniqueName: \"kubernetes.io/projected/a5514e1f-1fb4-4158-85c8-f17fe2b94eb8-kube-api-access-477x5\") pod \"maas-default-gateway-openshift-default-694db55c87-kvws4\" (UID: \"a5514e1f-1fb4-4158-85c8-f17fe2b94eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:20:56.321637 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.320783 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5514e1f-1fb4-4158-85c8-f17fe2b94eb8-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-kvws4\" (UID: \"a5514e1f-1fb4-4158-85c8-f17fe2b94eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:20:56.321637 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.319608 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5514e1f-1fb4-4158-85c8-f17fe2b94eb8-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-kvws4\" (UID: \"a5514e1f-1fb4-4158-85c8-f17fe2b94eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:20:56.321637 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.320274 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5514e1f-1fb4-4158-85c8-f17fe2b94eb8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-kvws4\" (UID: \"a5514e1f-1fb4-4158-85c8-f17fe2b94eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:20:56.321637 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.321520 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5514e1f-1fb4-4158-85c8-f17fe2b94eb8-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-kvws4\" (UID: \"a5514e1f-1fb4-4158-85c8-f17fe2b94eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:20:56.321989 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.321782 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5514e1f-1fb4-4158-85c8-f17fe2b94eb8-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-kvws4\" (UID: \"a5514e1f-1fb4-4158-85c8-f17fe2b94eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:20:56.322281 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.322257 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5514e1f-1fb4-4158-85c8-f17fe2b94eb8-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-kvws4\" (UID: \"a5514e1f-1fb4-4158-85c8-f17fe2b94eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:20:56.328192 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.324341 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5514e1f-1fb4-4158-85c8-f17fe2b94eb8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-kvws4\" (UID: \"a5514e1f-1fb4-4158-85c8-f17fe2b94eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:20:56.329309 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.329284 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5514e1f-1fb4-4158-85c8-f17fe2b94eb8-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-kvws4\" (UID: \"a5514e1f-1fb4-4158-85c8-f17fe2b94eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:20:56.400108 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.400054 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-477x5\" (UniqueName: \"kubernetes.io/projected/a5514e1f-1fb4-4158-85c8-f17fe2b94eb8-kube-api-access-477x5\") pod \"maas-default-gateway-openshift-default-694db55c87-kvws4\" (UID: \"a5514e1f-1fb4-4158-85c8-f17fe2b94eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:20:56.400564 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.400536 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5514e1f-1fb4-4158-85c8-f17fe2b94eb8-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-kvws4\" (UID: \"a5514e1f-1fb4-4158-85c8-f17fe2b94eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:20:56.516823 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.516723 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:20:56.988487 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.988207 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4"] Nov 03 00:20:56.988772 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.988727 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 00:20:56.988907 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.988806 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 00:20:56.988907 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:56.988854 5727 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 00:20:57.814344 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:57.814303 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" event={"ID":"a5514e1f-1fb4-4158-85c8-f17fe2b94eb8","Type":"ContainerStarted","Data":"b492fc7f2127b12e890e0f525ef79e57587842b03e7b96dc8955ca0b65729004"} Nov 03 00:20:57.814836 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:57.814360 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" event={"ID":"a5514e1f-1fb4-4158-85c8-f17fe2b94eb8","Type":"ContainerStarted","Data":"0143edbf16a15d0ec4cdd24e7d2e9eccc56f335cdd99802e912b109817bc8198"} Nov 03 00:20:57.845213 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:20:57.845181 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:20:57.944168 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:57.944096 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" podStartSLOduration=1.944082375 podStartE2EDuration="1.944082375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:57.941648309 +0000 UTC m=+5544.180145979" watchObservedRunningTime="2026-11-03 00:20:57.944082375 +0000 UTC m=+5544.182580037" Nov 03 00:20:58.518156 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:58.518108 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:20:58.519879 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:58.519853 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-kvws4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 03 00:20:58.520012 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:58.519900 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" podUID="a5514e1f-1fb4-4158-85c8-f17fe2b94eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 03 00:20:59.517496 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:59.517457 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-kvws4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 03 00:20:59.517931 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:20:59.517524 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" podUID="a5514e1f-1fb4-4158-85c8-f17fe2b94eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 03 00:21:00.517809 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:00.517774 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-kvws4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 03 00:21:00.518223 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:00.517829 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" podUID="a5514e1f-1fb4-4158-85c8-f17fe2b94eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 03 00:21:01.475892 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:01.475833 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-l67sb"] Nov 03 00:21:01.480027 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:01.480005 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-l67sb" Nov 03 00:21:01.518191 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:01.518168 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-kvws4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 03 00:21:01.518520 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:01.518218 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" podUID="a5514e1f-1fb4-4158-85c8-f17fe2b94eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 03 00:21:01.534956 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:01.534935 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-l67sb"] Nov 03 00:21:01.581105 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:01.581066 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:21:01.581294 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:01.581167 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:21:01.581294 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:01.581207 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:21:01.581497 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:01.581464 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:21:01.596776 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:01.596747 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54zz5\" (UniqueName: \"kubernetes.io/projected/9db58498-2a5d-4971-9b2f-920a0a954395-kube-api-access-54zz5\") pod \"maas-api-revoke-sub-f03629db-l67sb\" (UID: \"9db58498-2a5d-4971-9b2f-920a0a954395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-l67sb" Nov 03 00:21:01.596776 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:01.596782 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9db58498-2a5d-4971-9b2f-920a0a954395-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-l67sb\" (UID: \"9db58498-2a5d-4971-9b2f-920a0a954395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-l67sb" Nov 03 00:21:01.596946 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:01.596872 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9db58498-2a5d-4971-9b2f-920a0a954395-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-l67sb\" (UID: \"9db58498-2a5d-4971-9b2f-920a0a954395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-l67sb" Nov 03 00:21:01.698136 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:01.698099 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54zz5\" (UniqueName: \"kubernetes.io/projected/9db58498-2a5d-4971-9b2f-920a0a954395-kube-api-access-54zz5\") pod \"maas-api-revoke-sub-f03629db-l67sb\" (UID: \"9db58498-2a5d-4971-9b2f-920a0a954395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-l67sb" Nov 03 00:21:01.698324 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:01.698182 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9db58498-2a5d-4971-9b2f-920a0a954395-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-l67sb\" (UID: \"9db58498-2a5d-4971-9b2f-920a0a954395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-l67sb" Nov 03 00:21:01.698324 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:01.698254 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9db58498-2a5d-4971-9b2f-920a0a954395-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-l67sb\" (UID: \"9db58498-2a5d-4971-9b2f-920a0a954395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-l67sb" Nov 03 00:21:01.698932 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:01.698911 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9db58498-2a5d-4971-9b2f-920a0a954395-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-l67sb\" (UID: \"9db58498-2a5d-4971-9b2f-920a0a954395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-l67sb" Nov 03 00:21:01.700865 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:01.700839 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9db58498-2a5d-4971-9b2f-920a0a954395-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-l67sb\" (UID: \"9db58498-2a5d-4971-9b2f-920a0a954395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-l67sb" Nov 03 00:21:01.777746 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:01.777677 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54zz5\" (UniqueName: \"kubernetes.io/projected/9db58498-2a5d-4971-9b2f-920a0a954395-kube-api-access-54zz5\") pod \"maas-api-revoke-sub-f03629db-l67sb\" (UID: \"9db58498-2a5d-4971-9b2f-920a0a954395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-l67sb" Nov 03 00:21:01.794468 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:01.794443 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-l67sb" Nov 03 00:21:02.186634 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:02.186603 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-l67sb"] Nov 03 00:21:02.188510 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:21:02.188481 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db58498_2a5d_4971_9b2f_920a0a954395.slice/crio-c80184e23d203878a01955dc50dd3996e0dbbac456e3bfc754a5fcf52b3139a1 WatchSource:0}: Error finding container c80184e23d203878a01955dc50dd3996e0dbbac456e3bfc754a5fcf52b3139a1: Status 404 returned error can't find the container with id c80184e23d203878a01955dc50dd3996e0dbbac456e3bfc754a5fcf52b3139a1 Nov 03 00:21:02.517418 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:02.517387 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-kvws4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 03 00:21:02.517570 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:02.517438 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" podUID="a5514e1f-1fb4-4158-85c8-f17fe2b94eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 03 00:21:02.846629 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:02.846585 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-l67sb" event={"ID":"9db58498-2a5d-4971-9b2f-920a0a954395","Type":"ContainerStarted","Data":"3dc1fe862db49f451de5da739d56280bd5ed31d52452c6882e106fc35c1dad32"} Nov 03 00:21:02.847117 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:02.846650 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-l67sb" event={"ID":"9db58498-2a5d-4971-9b2f-920a0a954395","Type":"ContainerStarted","Data":"c80184e23d203878a01955dc50dd3996e0dbbac456e3bfc754a5fcf52b3139a1"} Nov 03 00:21:02.962762 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:02.962707 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-l67sb" podStartSLOduration=1.962692585 podStartE2EDuration="1.962692585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:02.961290036 +0000 UTC m=+5549.199787742" watchObservedRunningTime="2026-11-03 00:21:02.962692585 +0000 UTC m=+5549.201190261" Nov 03 00:21:03.517598 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:03.517516 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-kvws4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 03 00:21:03.517598 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:03.517575 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" podUID="a5514e1f-1fb4-4158-85c8-f17fe2b94eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 03 00:21:03.854782 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:03.854745 5727 generic.go:356] "Generic (PLEG): container finished" podID="9db58498-2a5d-4971-9b2f-920a0a954395" containerID="3dc1fe862db49f451de5da739d56280bd5ed31d52452c6882e106fc35c1dad32" exitCode=0 Nov 03 00:21:03.855196 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:03.854829 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-l67sb" event={"ID":"9db58498-2a5d-4971-9b2f-920a0a954395","Type":"ContainerDied","Data":"3dc1fe862db49f451de5da739d56280bd5ed31d52452c6882e106fc35c1dad32"} Nov 03 00:21:04.517752 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:04.517710 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-kvws4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 03 00:21:04.517942 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:04.517781 5727 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" podUID="a5514e1f-1fb4-4158-85c8-f17fe2b94eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 03 00:21:05.049415 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:05.049377 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-l67sb" Nov 03 00:21:05.135026 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:05.134940 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9db58498-2a5d-4971-9b2f-920a0a954395-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9db58498-2a5d-4971-9b2f-920a0a954395-maas-api-openshift-ca\") pod \"9db58498-2a5d-4971-9b2f-920a0a954395\" (UID: \"9db58498-2a5d-4971-9b2f-920a0a954395\") " Nov 03 00:21:05.135229 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:05.135067 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9db58498-2a5d-4971-9b2f-920a0a954395-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9db58498-2a5d-4971-9b2f-920a0a954395-maas-api-xks-ca\") pod \"9db58498-2a5d-4971-9b2f-920a0a954395\" (UID: \"9db58498-2a5d-4971-9b2f-920a0a954395\") " Nov 03 00:21:05.135229 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:05.135164 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db58498-2a5d-4971-9b2f-920a0a954395-kube-api-access-54zz5\" (UniqueName: \"kubernetes.io/projected/9db58498-2a5d-4971-9b2f-920a0a954395-kube-api-access-54zz5\") pod \"9db58498-2a5d-4971-9b2f-920a0a954395\" (UID: \"9db58498-2a5d-4971-9b2f-920a0a954395\") " Nov 03 00:21:05.135492 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:05.135463 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db58498-2a5d-4971-9b2f-920a0a954395-maas-api-openshift-ca" pod "9db58498-2a5d-4971-9b2f-920a0a954395" (UID: "9db58498-2a5d-4971-9b2f-920a0a954395"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:21:05.137864 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:05.137822 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db58498-2a5d-4971-9b2f-920a0a954395-kube-api-access-54zz5" pod "9db58498-2a5d-4971-9b2f-920a0a954395" (UID: "9db58498-2a5d-4971-9b2f-920a0a954395"). InnerVolumeSpecName "kube-api-access-54zz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:21:05.138009 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:05.137934 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9db58498-2a5d-4971-9b2f-920a0a954395-maas-api-xks-ca" pod "9db58498-2a5d-4971-9b2f-920a0a954395" (UID: "9db58498-2a5d-4971-9b2f-920a0a954395"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:21:05.236423 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:05.236380 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54zz5\" (UniqueName: \"kubernetes.io/projected/9db58498-2a5d-4971-9b2f-920a0a954395-kube-api-access-54zz5\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:21:05.236423 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:05.236417 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9db58498-2a5d-4971-9b2f-920a0a954395-maas-api-openshift-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:21:05.236601 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:05.236434 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9db58498-2a5d-4971-9b2f-920a0a954395-maas-api-xks-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:21:05.525333 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:05.525227 5727 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:21:05.526044 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:05.525991 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:21:05.526751 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:05.526725 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" Nov 03 00:21:05.868816 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:05.868787 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-l67sb" Nov 03 00:21:05.868979 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:05.868815 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-l67sb" event={"ID":"9db58498-2a5d-4971-9b2f-920a0a954395","Type":"ContainerDied","Data":"c80184e23d203878a01955dc50dd3996e0dbbac456e3bfc754a5fcf52b3139a1"} Nov 03 00:21:05.868979 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:05.868865 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c80184e23d203878a01955dc50dd3996e0dbbac456e3bfc754a5fcf52b3139a1" Nov 03 00:21:21.651142 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:21.651099 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tfwm9"] Nov 03 00:21:21.651838 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:21.651815 5727 cpu_manager.go:414] "containerMap: removing container" podUID="9db58498-2a5d-4971-9b2f-920a0a954395" containerName="revoke-keys" Nov 03 00:21:21.651933 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:21.651841 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9db58498-2a5d-4971-9b2f-920a0a954395" containerName="revoke-keys" Nov 03 00:21:21.652017 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:21.652002 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="9db58498-2a5d-4971-9b2f-920a0a954395" containerName="revoke-keys" Nov 03 00:21:21.654909 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:21.654887 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tfwm9" Nov 03 00:21:21.758976 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:21.758942 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:21:21.758976 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:21.758962 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:21:21.758976 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:21.758978 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:21:21.759238 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:21.759011 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:21:21.761741 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:21.761722 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tfwm9"] Nov 03 00:21:21.835099 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:21.835073 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21d05897-70cb-4441-abba-1e7136943153-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-tfwm9\" (UID: \"21d05897-70cb-4441-abba-1e7136943153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tfwm9" Nov 03 00:21:21.835099 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:21.835100 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjzts\" (UniqueName: \"kubernetes.io/projected/21d05897-70cb-4441-abba-1e7136943153-kube-api-access-kjzts\") pod \"maas-api-revoke-sub-0165d7d2-tfwm9\" (UID: \"21d05897-70cb-4441-abba-1e7136943153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tfwm9" Nov 03 00:21:21.835358 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:21.835287 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21d05897-70cb-4441-abba-1e7136943153-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-tfwm9\" (UID: \"21d05897-70cb-4441-abba-1e7136943153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tfwm9" Nov 03 00:21:21.936556 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:21.936486 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21d05897-70cb-4441-abba-1e7136943153-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-tfwm9\" (UID: \"21d05897-70cb-4441-abba-1e7136943153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tfwm9" Nov 03 00:21:21.936556 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:21.936517 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjzts\" (UniqueName: \"kubernetes.io/projected/21d05897-70cb-4441-abba-1e7136943153-kube-api-access-kjzts\") pod \"maas-api-revoke-sub-0165d7d2-tfwm9\" (UID: \"21d05897-70cb-4441-abba-1e7136943153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tfwm9" Nov 03 00:21:21.936743 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:21.936578 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21d05897-70cb-4441-abba-1e7136943153-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-tfwm9\" (UID: \"21d05897-70cb-4441-abba-1e7136943153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tfwm9" Nov 03 00:21:21.937200 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:21.937179 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21d05897-70cb-4441-abba-1e7136943153-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-tfwm9\" (UID: \"21d05897-70cb-4441-abba-1e7136943153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tfwm9" Nov 03 00:21:21.939042 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:21.939020 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21d05897-70cb-4441-abba-1e7136943153-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-tfwm9\" (UID: \"21d05897-70cb-4441-abba-1e7136943153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tfwm9" Nov 03 00:21:21.997269 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:21.997247 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjzts\" (UniqueName: \"kubernetes.io/projected/21d05897-70cb-4441-abba-1e7136943153-kube-api-access-kjzts\") pod \"maas-api-revoke-sub-0165d7d2-tfwm9\" (UID: \"21d05897-70cb-4441-abba-1e7136943153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tfwm9" Nov 03 00:21:22.268538 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:22.268441 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tfwm9" Nov 03 00:21:22.645892 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:22.645862 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tfwm9"] Nov 03 00:21:22.647208 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:21:22.647122 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d05897_70cb_4441_abba_1e7136943153.slice/crio-7882eea05b579df0733684e418054d46ee5a422246b9e4ee1e4b55a7d612c16b WatchSource:0}: Error finding container 7882eea05b579df0733684e418054d46ee5a422246b9e4ee1e4b55a7d612c16b: Status 404 returned error can't find the container with id 7882eea05b579df0733684e418054d46ee5a422246b9e4ee1e4b55a7d612c16b Nov 03 00:21:22.999630 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:22.999595 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tfwm9" event={"ID":"21d05897-70cb-4441-abba-1e7136943153","Type":"ContainerStarted","Data":"4c6d81eb03519e30c124eab1f8578a79471ff56eaf4c342feb5e4a688cec9c22"} Nov 03 00:21:22.999956 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:22.999657 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tfwm9" event={"ID":"21d05897-70cb-4441-abba-1e7136943153","Type":"ContainerStarted","Data":"7882eea05b579df0733684e418054d46ee5a422246b9e4ee1e4b55a7d612c16b"} Nov 03 00:21:23.115753 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:23.115691 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tfwm9" podStartSLOduration=2.115678544 podStartE2EDuration="2.115678544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:23.114043784 +0000 UTC m=+5569.352541459" watchObservedRunningTime="2026-11-03 00:21:23.115678544 +0000 UTC m=+5569.354176234" Nov 03 00:21:24.007159 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:24.007099 5727 generic.go:356] "Generic (PLEG): container finished" podID="21d05897-70cb-4441-abba-1e7136943153" containerID="4c6d81eb03519e30c124eab1f8578a79471ff56eaf4c342feb5e4a688cec9c22" exitCode=0 Nov 03 00:21:24.007623 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:24.007195 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tfwm9" event={"ID":"21d05897-70cb-4441-abba-1e7136943153","Type":"ContainerDied","Data":"4c6d81eb03519e30c124eab1f8578a79471ff56eaf4c342feb5e4a688cec9c22"} Nov 03 00:21:25.156016 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:25.155994 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tfwm9" Nov 03 00:21:25.165935 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:25.165908 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21d05897-70cb-4441-abba-1e7136943153-kube-api-access-kjzts\" (UniqueName: \"kubernetes.io/projected/21d05897-70cb-4441-abba-1e7136943153-kube-api-access-kjzts\") pod \"21d05897-70cb-4441-abba-1e7136943153\" (UID: \"21d05897-70cb-4441-abba-1e7136943153\") " Nov 03 00:21:25.166164 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:25.166143 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21d05897-70cb-4441-abba-1e7136943153-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21d05897-70cb-4441-abba-1e7136943153-maas-api-openshift-ca\") pod \"21d05897-70cb-4441-abba-1e7136943153\" (UID: \"21d05897-70cb-4441-abba-1e7136943153\") " Nov 03 00:21:25.166226 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:25.166209 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21d05897-70cb-4441-abba-1e7136943153-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21d05897-70cb-4441-abba-1e7136943153-maas-api-xks-ca\") pod \"21d05897-70cb-4441-abba-1e7136943153\" (UID: \"21d05897-70cb-4441-abba-1e7136943153\") " Nov 03 00:21:25.166458 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:25.166431 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21d05897-70cb-4441-abba-1e7136943153-maas-api-openshift-ca" pod "21d05897-70cb-4441-abba-1e7136943153" (UID: "21d05897-70cb-4441-abba-1e7136943153"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:21:25.166664 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:25.166645 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21d05897-70cb-4441-abba-1e7136943153-maas-api-openshift-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:21:25.168303 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:25.168277 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21d05897-70cb-4441-abba-1e7136943153-kube-api-access-kjzts" pod "21d05897-70cb-4441-abba-1e7136943153" (UID: "21d05897-70cb-4441-abba-1e7136943153"). InnerVolumeSpecName "kube-api-access-kjzts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:21:25.168398 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:25.168319 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21d05897-70cb-4441-abba-1e7136943153-maas-api-xks-ca" pod "21d05897-70cb-4441-abba-1e7136943153" (UID: "21d05897-70cb-4441-abba-1e7136943153"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:21:25.267950 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:25.267874 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21d05897-70cb-4441-abba-1e7136943153-maas-api-xks-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:21:25.267950 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:25.267909 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjzts\" (UniqueName: \"kubernetes.io/projected/21d05897-70cb-4441-abba-1e7136943153-kube-api-access-kjzts\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:21:26.020456 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:26.020430 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tfwm9" Nov 03 00:21:26.020643 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:26.020424 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tfwm9" event={"ID":"21d05897-70cb-4441-abba-1e7136943153","Type":"ContainerDied","Data":"7882eea05b579df0733684e418054d46ee5a422246b9e4ee1e4b55a7d612c16b"} Nov 03 00:21:26.020643 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:26.020551 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7882eea05b579df0733684e418054d46ee5a422246b9e4ee1e4b55a7d612c16b" Nov 03 00:21:27.846610 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:21:27.846573 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:21:33.366355 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:33.366318 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6l5zs"] Nov 03 00:21:33.366778 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:33.366764 5727 cpu_manager.go:414] "containerMap: removing container" podUID="21d05897-70cb-4441-abba-1e7136943153" containerName="revoke-keys" Nov 03 00:21:33.366833 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:33.366780 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21d05897-70cb-4441-abba-1e7136943153" containerName="revoke-keys" Nov 03 00:21:33.366889 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:33.366879 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="21d05897-70cb-4441-abba-1e7136943153" containerName="revoke-keys" Nov 03 00:21:33.370352 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:33.370331 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6l5zs" Nov 03 00:21:33.424026 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:33.423985 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:21:33.450470 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:33.450437 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfd98fd6-35e5-4ae5-b591-a11f991b4b18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-6l5zs\" (UID: \"cfd98fd6-35e5-4ae5-b591-a11f991b4b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6l5zs" Nov 03 00:21:33.450630 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:33.450584 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfd98fd6-35e5-4ae5-b591-a11f991b4b18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-6l5zs\" (UID: \"cfd98fd6-35e5-4ae5-b591-a11f991b4b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6l5zs" Nov 03 00:21:33.450695 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:33.450667 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwwg2\" (UniqueName: \"kubernetes.io/projected/cfd98fd6-35e5-4ae5-b591-a11f991b4b18-kube-api-access-mwwg2\") pod \"maas-api-revoke-sub-6b19338f-6l5zs\" (UID: \"cfd98fd6-35e5-4ae5-b591-a11f991b4b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6l5zs" Nov 03 00:21:33.473498 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:33.473465 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:21:33.473498 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:33.473489 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:21:33.473711 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:33.473469 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:21:33.476312 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:33.476289 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6l5zs"] Nov 03 00:21:33.552020 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:33.551989 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfd98fd6-35e5-4ae5-b591-a11f991b4b18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-6l5zs\" (UID: \"cfd98fd6-35e5-4ae5-b591-a11f991b4b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6l5zs" Nov 03 00:21:33.552226 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:33.552027 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwwg2\" (UniqueName: \"kubernetes.io/projected/cfd98fd6-35e5-4ae5-b591-a11f991b4b18-kube-api-access-mwwg2\") pod \"maas-api-revoke-sub-6b19338f-6l5zs\" (UID: \"cfd98fd6-35e5-4ae5-b591-a11f991b4b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6l5zs" Nov 03 00:21:33.552226 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:33.552087 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfd98fd6-35e5-4ae5-b591-a11f991b4b18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-6l5zs\" (UID: \"cfd98fd6-35e5-4ae5-b591-a11f991b4b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6l5zs" Nov 03 00:21:33.552832 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:33.552750 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfd98fd6-35e5-4ae5-b591-a11f991b4b18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-6l5zs\" (UID: \"cfd98fd6-35e5-4ae5-b591-a11f991b4b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6l5zs" Nov 03 00:21:33.554605 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:33.554580 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfd98fd6-35e5-4ae5-b591-a11f991b4b18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-6l5zs\" (UID: \"cfd98fd6-35e5-4ae5-b591-a11f991b4b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6l5zs" Nov 03 00:21:33.612849 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:33.612823 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwwg2\" (UniqueName: \"kubernetes.io/projected/cfd98fd6-35e5-4ae5-b591-a11f991b4b18-kube-api-access-mwwg2\") pod \"maas-api-revoke-sub-6b19338f-6l5zs\" (UID: \"cfd98fd6-35e5-4ae5-b591-a11f991b4b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6l5zs" Nov 03 00:21:33.688034 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:33.687979 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6l5zs" Nov 03 00:21:34.064812 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:34.064784 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6l5zs"] Nov 03 00:21:34.066313 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:21:34.066270 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd98fd6_35e5_4ae5_b591_a11f991b4b18.slice/crio-80142c0305be27b49cba2ddcb5e6b94334925c4f0cf9cf8688a35724030cd5a9 WatchSource:0}: Error finding container 80142c0305be27b49cba2ddcb5e6b94334925c4f0cf9cf8688a35724030cd5a9: Status 404 returned error can't find the container with id 80142c0305be27b49cba2ddcb5e6b94334925c4f0cf9cf8688a35724030cd5a9 Nov 03 00:21:35.081552 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:35.081510 5727 generic.go:356] "Generic (PLEG): container finished" podID="cfd98fd6-35e5-4ae5-b591-a11f991b4b18" containerID="694f35a21e661c78ffd5e2ee142a028788c5fb1b334d7f4ba76264df0bf9c305" exitCode=0 Nov 03 00:21:35.082062 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:35.081576 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6l5zs" event={"ID":"cfd98fd6-35e5-4ae5-b591-a11f991b4b18","Type":"ContainerDied","Data":"694f35a21e661c78ffd5e2ee142a028788c5fb1b334d7f4ba76264df0bf9c305"} Nov 03 00:21:35.082062 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:35.081677 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6l5zs" event={"ID":"cfd98fd6-35e5-4ae5-b591-a11f991b4b18","Type":"ContainerStarted","Data":"80142c0305be27b49cba2ddcb5e6b94334925c4f0cf9cf8688a35724030cd5a9"} Nov 03 00:21:36.000470 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:36.000429 5727 scope.go:122] "RemoveContainer" containerID="6f0a16ef00fdafe2a974c2645315cea97d8e756211614b4ba2f80157444fdb08" Nov 03 00:21:36.231917 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:36.231888 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6l5zs" Nov 03 00:21:36.277058 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:36.276998 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfd98fd6-35e5-4ae5-b591-a11f991b4b18-kube-api-access-mwwg2\" (UniqueName: \"kubernetes.io/projected/cfd98fd6-35e5-4ae5-b591-a11f991b4b18-kube-api-access-mwwg2\") pod \"cfd98fd6-35e5-4ae5-b591-a11f991b4b18\" (UID: \"cfd98fd6-35e5-4ae5-b591-a11f991b4b18\") " Nov 03 00:21:36.277058 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:36.277037 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfd98fd6-35e5-4ae5-b591-a11f991b4b18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfd98fd6-35e5-4ae5-b591-a11f991b4b18-maas-api-xks-ca\") pod \"cfd98fd6-35e5-4ae5-b591-a11f991b4b18\" (UID: \"cfd98fd6-35e5-4ae5-b591-a11f991b4b18\") " Nov 03 00:21:36.277265 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:36.277065 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfd98fd6-35e5-4ae5-b591-a11f991b4b18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfd98fd6-35e5-4ae5-b591-a11f991b4b18-maas-api-openshift-ca\") pod \"cfd98fd6-35e5-4ae5-b591-a11f991b4b18\" (UID: \"cfd98fd6-35e5-4ae5-b591-a11f991b4b18\") " Nov 03 00:21:36.277451 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:36.277428 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfd98fd6-35e5-4ae5-b591-a11f991b4b18-maas-api-openshift-ca" pod "cfd98fd6-35e5-4ae5-b591-a11f991b4b18" (UID: "cfd98fd6-35e5-4ae5-b591-a11f991b4b18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:21:36.279379 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:36.279357 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfd98fd6-35e5-4ae5-b591-a11f991b4b18-maas-api-xks-ca" pod "cfd98fd6-35e5-4ae5-b591-a11f991b4b18" (UID: "cfd98fd6-35e5-4ae5-b591-a11f991b4b18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:21:36.279457 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:36.279419 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd98fd6-35e5-4ae5-b591-a11f991b4b18-kube-api-access-mwwg2" pod "cfd98fd6-35e5-4ae5-b591-a11f991b4b18" (UID: "cfd98fd6-35e5-4ae5-b591-a11f991b4b18"). InnerVolumeSpecName "kube-api-access-mwwg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:21:36.377984 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:36.377952 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwwg2\" (UniqueName: \"kubernetes.io/projected/cfd98fd6-35e5-4ae5-b591-a11f991b4b18-kube-api-access-mwwg2\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:21:36.377984 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:36.377979 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfd98fd6-35e5-4ae5-b591-a11f991b4b18-maas-api-xks-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:21:36.377984 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:36.377989 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfd98fd6-35e5-4ae5-b591-a11f991b4b18-maas-api-openshift-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:21:37.095664 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:37.095632 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6l5zs" Nov 03 00:21:37.095831 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:37.095669 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6l5zs" event={"ID":"cfd98fd6-35e5-4ae5-b591-a11f991b4b18","Type":"ContainerDied","Data":"80142c0305be27b49cba2ddcb5e6b94334925c4f0cf9cf8688a35724030cd5a9"} Nov 03 00:21:37.095831 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:37.095717 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80142c0305be27b49cba2ddcb5e6b94334925c4f0cf9cf8688a35724030cd5a9" Nov 03 00:21:48.087207 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:48.084943 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj"] Nov 03 00:21:48.087207 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:48.085673 5727 cpu_manager.go:414] "containerMap: removing container" podUID="cfd98fd6-35e5-4ae5-b591-a11f991b4b18" containerName="revoke-keys" Nov 03 00:21:48.087207 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:48.085693 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfd98fd6-35e5-4ae5-b591-a11f991b4b18" containerName="revoke-keys" Nov 03 00:21:48.087207 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:48.085865 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfd98fd6-35e5-4ae5-b591-a11f991b4b18" containerName="revoke-keys" Nov 03 00:21:48.104665 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:48.103760 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj" Nov 03 00:21:48.139702 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:48.138790 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj"] Nov 03 00:21:48.189861 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:48.189724 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-37a87c71-kserve-self-signed-certs\"" Nov 03 00:21:48.189861 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:48.189819 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-37a87c71-epp-sa-dockercfg-hzjvp\"" Nov 03 00:21:48.233773 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:48.233556 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c62d1a0f-fe74-4d63-bdd9-e87aaa96435a-tls-certs\") pod \"e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj\" (UID: \"c62d1a0f-fe74-4d63-bdd9-e87aaa96435a\") " pod="llm/e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj" Nov 03 00:21:48.233773 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:48.233626 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ffg6\" (UniqueName: \"kubernetes.io/projected/c62d1a0f-fe74-4d63-bdd9-e87aaa96435a-kube-api-access-9ffg6\") pod \"e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj\" (UID: \"c62d1a0f-fe74-4d63-bdd9-e87aaa96435a\") " pod="llm/e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj" Nov 03 00:21:48.334777 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:48.334735 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c62d1a0f-fe74-4d63-bdd9-e87aaa96435a-tls-certs\") pod \"e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj\" (UID: \"c62d1a0f-fe74-4d63-bdd9-e87aaa96435a\") " pod="llm/e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj" Nov 03 00:21:48.334777 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:48.334784 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ffg6\" (UniqueName: \"kubernetes.io/projected/c62d1a0f-fe74-4d63-bdd9-e87aaa96435a-kube-api-access-9ffg6\") pod \"e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj\" (UID: \"c62d1a0f-fe74-4d63-bdd9-e87aaa96435a\") " pod="llm/e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj" Nov 03 00:21:48.338180 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:48.338095 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c62d1a0f-fe74-4d63-bdd9-e87aaa96435a-tls-certs\") pod \"e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj\" (UID: \"c62d1a0f-fe74-4d63-bdd9-e87aaa96435a\") " pod="llm/e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj" Nov 03 00:21:48.396870 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:48.396793 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ffg6\" (UniqueName: \"kubernetes.io/projected/c62d1a0f-fe74-4d63-bdd9-e87aaa96435a-kube-api-access-9ffg6\") pod \"e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj\" (UID: \"c62d1a0f-fe74-4d63-bdd9-e87aaa96435a\") " pod="llm/e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj" Nov 03 00:21:48.444464 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:48.444436 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj" Nov 03 00:21:48.926372 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:48.919007 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj"] Nov 03 00:21:49.223016 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:49.222934 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj" event={"ID":"c62d1a0f-fe74-4d63-bdd9-e87aaa96435a","Type":"ContainerStarted","Data":"10da86acf4f018407c7a6434f008fa76eda0f2d52609101ff4f1cfbb6fd2d9b9"} Nov 03 00:21:51.240699 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:51.240615 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj" event={"ID":"c62d1a0f-fe74-4d63-bdd9-e87aaa96435a","Type":"ContainerStarted","Data":"ee00c4ec08884e84030593b989b2b1eb4d40d40da11bdf94d8f6fc59f183a70a"} Nov 03 00:21:51.241064 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:51.240708 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj" Nov 03 00:21:51.356850 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:21:51.356799 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj" podStartSLOduration=1.388760889 podStartE2EDuration="3.356785793s" totalImagesPullingTime="1.968024904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:21:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:51.355903672 +0000 UTC m=+5597.594401358" watchObservedRunningTime="2026-11-03 00:21:51.356785793 +0000 UTC m=+5597.595283446" Nov 03 00:21:57.849427 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:21:57.849389 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:22:22.249822 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:22:22.249796 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj" Nov 03 00:22:27.846231 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:22:27.846195 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:22:36.025096 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:22:36.025062 5727 scope.go:122] "RemoveContainer" containerID="26874028487ea372c2be868bed79a95dc2cad67a28ab66e89ee29bb8e08ba1e7" Nov 03 00:22:36.035027 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:22:36.035006 5727 scope.go:122] "RemoveContainer" containerID="fbc7858dbe5fccf479eada8e3f7c0d4e7b8d7f5e43c8e5794eb363ea0bd64800" Nov 03 00:22:56.179928 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:22:56.179893 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj"] Nov 03 00:22:57.846201 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:22:57.846156 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:23:02.252250 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:23:02.252207 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj" podUID="c62d1a0f-fe74-4d63-bdd9-e87aaa96435a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:23:11.202557 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:23:11.202515 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj" podUID="c62d1a0f-fe74-4d63-bdd9-e87aaa96435a" containerName="main" containerID="cri-o://ee00c4ec08884e84030593b989b2b1eb4d40d40da11bdf94d8f6fc59f183a70a" gracePeriod=45 Nov 03 00:23:12.249640 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:23:12.249599 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj" podUID="c62d1a0f-fe74-4d63-bdd9-e87aaa96435a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:23:22.249242 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:23:22.249207 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj" podUID="c62d1a0f-fe74-4d63-bdd9-e87aaa96435a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:23:22.249633 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:23:22.249306 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj" Nov 03 00:23:27.844884 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:23:27.844847 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:23:32.249763 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:23:32.249723 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj" podUID="c62d1a0f-fe74-4d63-bdd9-e87aaa96435a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:23:35.110976 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:23:35.110946 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 03 00:23:35.127050 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:23:35.127026 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 03 00:23:35.229738 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:23:35.229712 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 03 00:23:35.244395 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:23:35.244365 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 03 00:23:41.467907 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:23:41.467847 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj" Nov 03 00:23:41.633801 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:23:41.633772 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c62d1a0f-fe74-4d63-bdd9-e87aaa96435a-tls-certs\" (UniqueName: \"kubernetes.io/secret/c62d1a0f-fe74-4d63-bdd9-e87aaa96435a-tls-certs\") pod \"c62d1a0f-fe74-4d63-bdd9-e87aaa96435a\" (UID: \"c62d1a0f-fe74-4d63-bdd9-e87aaa96435a\") " Nov 03 00:23:41.633973 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:23:41.633831 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c62d1a0f-fe74-4d63-bdd9-e87aaa96435a-kube-api-access-9ffg6\" (UniqueName: \"kubernetes.io/projected/c62d1a0f-fe74-4d63-bdd9-e87aaa96435a-kube-api-access-9ffg6\") pod \"c62d1a0f-fe74-4d63-bdd9-e87aaa96435a\" (UID: \"c62d1a0f-fe74-4d63-bdd9-e87aaa96435a\") " Nov 03 00:23:41.636307 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:23:41.636275 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c62d1a0f-fe74-4d63-bdd9-e87aaa96435a-tls-certs" pod "c62d1a0f-fe74-4d63-bdd9-e87aaa96435a" (UID: "c62d1a0f-fe74-4d63-bdd9-e87aaa96435a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:23:41.636307 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:23:41.636276 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c62d1a0f-fe74-4d63-bdd9-e87aaa96435a-kube-api-access-9ffg6" pod "c62d1a0f-fe74-4d63-bdd9-e87aaa96435a" (UID: "c62d1a0f-fe74-4d63-bdd9-e87aaa96435a"). InnerVolumeSpecName "kube-api-access-9ffg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:23:41.735524 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:23:41.735450 5727 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c62d1a0f-fe74-4d63-bdd9-e87aaa96435a-tls-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:41.735524 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:23:41.735477 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ffg6\" (UniqueName: \"kubernetes.io/projected/c62d1a0f-fe74-4d63-bdd9-e87aaa96435a-kube-api-access-9ffg6\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:42.006788 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:23:42.006704 5727 generic.go:356] "Generic (PLEG): container finished" podID="c62d1a0f-fe74-4d63-bdd9-e87aaa96435a" containerID="ee00c4ec08884e84030593b989b2b1eb4d40d40da11bdf94d8f6fc59f183a70a" exitCode=0 Nov 03 00:23:42.006788 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:23:42.006750 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj" Nov 03 00:23:42.007098 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:23:42.006792 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj" event={"ID":"c62d1a0f-fe74-4d63-bdd9-e87aaa96435a","Type":"ContainerDied","Data":"ee00c4ec08884e84030593b989b2b1eb4d40d40da11bdf94d8f6fc59f183a70a"} Nov 03 00:23:42.007098 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:23:42.006846 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj" event={"ID":"c62d1a0f-fe74-4d63-bdd9-e87aaa96435a","Type":"ContainerDied","Data":"10da86acf4f018407c7a6434f008fa76eda0f2d52609101ff4f1cfbb6fd2d9b9"} Nov 03 00:23:42.007098 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:23:42.006871 5727 scope.go:122] "RemoveContainer" containerID="ee00c4ec08884e84030593b989b2b1eb4d40d40da11bdf94d8f6fc59f183a70a" Nov 03 00:23:42.016619 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:23:42.016598 5727 scope.go:122] "RemoveContainer" containerID="ee00c4ec08884e84030593b989b2b1eb4d40d40da11bdf94d8f6fc59f183a70a" Nov 03 00:23:42.016882 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:23:42.016862 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee00c4ec08884e84030593b989b2b1eb4d40d40da11bdf94d8f6fc59f183a70a\": container with ID starting with ee00c4ec08884e84030593b989b2b1eb4d40d40da11bdf94d8f6fc59f183a70a not found: ID does not exist" containerID="ee00c4ec08884e84030593b989b2b1eb4d40d40da11bdf94d8f6fc59f183a70a" Nov 03 00:23:42.016931 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:23:42.016889 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee00c4ec08884e84030593b989b2b1eb4d40d40da11bdf94d8f6fc59f183a70a"} err="failed to get container status \"ee00c4ec08884e84030593b989b2b1eb4d40d40da11bdf94d8f6fc59f183a70a\": rpc error: code = NotFound desc = could not find container \"ee00c4ec08884e84030593b989b2b1eb4d40d40da11bdf94d8f6fc59f183a70a\": container with ID starting with ee00c4ec08884e84030593b989b2b1eb4d40d40da11bdf94d8f6fc59f183a70a not found: ID does not exist" Nov 03 00:23:42.170051 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:23:42.170018 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj"] Nov 03 00:23:42.219496 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:23:42.219462 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-37a87c71-kserve-router-scheduler-77ff5c6589-xkcvj"] Nov 03 00:23:43.882603 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:23:43.882567 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c62d1a0f-fe74-4d63-bdd9-e87aaa96435a" path="/var/lib/kubelet/pods/c62d1a0f-fe74-4d63-bdd9-e87aaa96435a/volumes" Nov 03 00:23:57.845322 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:23:57.845279 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:24:27.845194 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:24:27.845152 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:24:37.825610 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:37.825573 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b823784f-6jtzt"] Nov 03 00:24:37.826049 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:37.826035 5727 cpu_manager.go:414] "containerMap: removing container" podUID="c62d1a0f-fe74-4d63-bdd9-e87aaa96435a" containerName="main" Nov 03 00:24:37.826104 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:37.826050 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c62d1a0f-fe74-4d63-bdd9-e87aaa96435a" containerName="main" Nov 03 00:24:37.826190 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:37.826180 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="c62d1a0f-fe74-4d63-bdd9-e87aaa96435a" containerName="main" Nov 03 00:24:37.830701 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:37.830685 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b823784f-6jtzt" Nov 03 00:24:37.854626 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:37.854595 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b14adfd8-b845-4a2d-bf35-f8c45da4343f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b823784f-6jtzt\" (UID: \"b14adfd8-b845-4a2d-bf35-f8c45da4343f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b823784f-6jtzt" Nov 03 00:24:37.854744 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:37.854653 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b14adfd8-b845-4a2d-bf35-f8c45da4343f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b823784f-6jtzt\" (UID: \"b14adfd8-b845-4a2d-bf35-f8c45da4343f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b823784f-6jtzt" Nov 03 00:24:37.854790 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:37.854773 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr7sx\" (UniqueName: \"kubernetes.io/projected/b14adfd8-b845-4a2d-bf35-f8c45da4343f-kube-api-access-gr7sx\") pod \"maas-api-revoke-sub-b823784f-6jtzt\" (UID: \"b14adfd8-b845-4a2d-bf35-f8c45da4343f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b823784f-6jtzt" Nov 03 00:24:37.883091 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:37.883066 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:24:37.932366 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:37.932339 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:24:37.932366 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:37.932353 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:24:37.932366 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:37.932343 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:24:37.935338 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:37.935313 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b823784f-6jtzt"] Nov 03 00:24:37.955858 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:37.955836 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b14adfd8-b845-4a2d-bf35-f8c45da4343f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b823784f-6jtzt\" (UID: \"b14adfd8-b845-4a2d-bf35-f8c45da4343f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b823784f-6jtzt" Nov 03 00:24:37.955963 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:37.955875 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b14adfd8-b845-4a2d-bf35-f8c45da4343f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b823784f-6jtzt\" (UID: \"b14adfd8-b845-4a2d-bf35-f8c45da4343f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b823784f-6jtzt" Nov 03 00:24:37.955963 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:37.955920 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr7sx\" (UniqueName: \"kubernetes.io/projected/b14adfd8-b845-4a2d-bf35-f8c45da4343f-kube-api-access-gr7sx\") pod \"maas-api-revoke-sub-b823784f-6jtzt\" (UID: \"b14adfd8-b845-4a2d-bf35-f8c45da4343f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b823784f-6jtzt" Nov 03 00:24:37.956578 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:37.956554 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b14adfd8-b845-4a2d-bf35-f8c45da4343f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b823784f-6jtzt\" (UID: \"b14adfd8-b845-4a2d-bf35-f8c45da4343f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b823784f-6jtzt" Nov 03 00:24:37.958537 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:37.958518 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b14adfd8-b845-4a2d-bf35-f8c45da4343f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b823784f-6jtzt\" (UID: \"b14adfd8-b845-4a2d-bf35-f8c45da4343f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b823784f-6jtzt" Nov 03 00:24:38.016596 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:38.016574 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr7sx\" (UniqueName: \"kubernetes.io/projected/b14adfd8-b845-4a2d-bf35-f8c45da4343f-kube-api-access-gr7sx\") pod \"maas-api-revoke-sub-b823784f-6jtzt\" (UID: \"b14adfd8-b845-4a2d-bf35-f8c45da4343f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b823784f-6jtzt" Nov 03 00:24:38.144113 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:38.144029 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b823784f-6jtzt" Nov 03 00:24:38.521171 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:38.521145 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b823784f-6jtzt"] Nov 03 00:24:38.522542 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:24:38.522516 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb14adfd8_b845_4a2d_bf35_f8c45da4343f.slice/crio-d6d418e812744a5f22a4111cc8d0b12ff3f699edec49f5772f291fa02f26aa6b WatchSource:0}: Error finding container d6d418e812744a5f22a4111cc8d0b12ff3f699edec49f5772f291fa02f26aa6b: Status 404 returned error can't find the container with id d6d418e812744a5f22a4111cc8d0b12ff3f699edec49f5772f291fa02f26aa6b Nov 03 00:24:39.369182 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:39.369145 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b823784f-6jtzt" event={"ID":"b14adfd8-b845-4a2d-bf35-f8c45da4343f","Type":"ContainerStarted","Data":"31a22d7224ef58ebcc0de602abcfd7e472d1e13156d474291f5827933b5660dc"} Nov 03 00:24:39.369587 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:39.369204 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b823784f-6jtzt" event={"ID":"b14adfd8-b845-4a2d-bf35-f8c45da4343f","Type":"ContainerStarted","Data":"d6d418e812744a5f22a4111cc8d0b12ff3f699edec49f5772f291fa02f26aa6b"} Nov 03 00:24:39.486171 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:39.486085 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b823784f-6jtzt" podStartSLOduration=2.486065254 podStartE2EDuration="2.486065254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:24:39.483703861 +0000 UTC m=+5765.722201542" watchObservedRunningTime="2026-11-03 00:24:39.486065254 +0000 UTC m=+5765.724562916" Nov 03 00:24:40.378221 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:40.378173 5727 generic.go:356] "Generic (PLEG): container finished" podID="b14adfd8-b845-4a2d-bf35-f8c45da4343f" containerID="31a22d7224ef58ebcc0de602abcfd7e472d1e13156d474291f5827933b5660dc" exitCode=0 Nov 03 00:24:40.378769 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:40.378268 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b823784f-6jtzt" event={"ID":"b14adfd8-b845-4a2d-bf35-f8c45da4343f","Type":"ContainerDied","Data":"31a22d7224ef58ebcc0de602abcfd7e472d1e13156d474291f5827933b5660dc"} Nov 03 00:24:41.550386 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:41.550357 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b823784f-6jtzt" Nov 03 00:24:41.629402 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:41.629367 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b14adfd8-b845-4a2d-bf35-f8c45da4343f-kube-api-access-gr7sx\" (UniqueName: \"kubernetes.io/projected/b14adfd8-b845-4a2d-bf35-f8c45da4343f-kube-api-access-gr7sx\") pod \"b14adfd8-b845-4a2d-bf35-f8c45da4343f\" (UID: \"b14adfd8-b845-4a2d-bf35-f8c45da4343f\") " Nov 03 00:24:41.629581 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:41.629413 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b14adfd8-b845-4a2d-bf35-f8c45da4343f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b14adfd8-b845-4a2d-bf35-f8c45da4343f-maas-api-openshift-ca\") pod \"b14adfd8-b845-4a2d-bf35-f8c45da4343f\" (UID: \"b14adfd8-b845-4a2d-bf35-f8c45da4343f\") " Nov 03 00:24:41.629581 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:41.629518 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b14adfd8-b845-4a2d-bf35-f8c45da4343f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b14adfd8-b845-4a2d-bf35-f8c45da4343f-maas-api-xks-ca\") pod \"b14adfd8-b845-4a2d-bf35-f8c45da4343f\" (UID: \"b14adfd8-b845-4a2d-bf35-f8c45da4343f\") " Nov 03 00:24:41.629858 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:41.629832 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b14adfd8-b845-4a2d-bf35-f8c45da4343f-maas-api-openshift-ca" pod "b14adfd8-b845-4a2d-bf35-f8c45da4343f" (UID: "b14adfd8-b845-4a2d-bf35-f8c45da4343f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:24:41.631742 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:41.631713 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b14adfd8-b845-4a2d-bf35-f8c45da4343f-kube-api-access-gr7sx" pod "b14adfd8-b845-4a2d-bf35-f8c45da4343f" (UID: "b14adfd8-b845-4a2d-bf35-f8c45da4343f"). InnerVolumeSpecName "kube-api-access-gr7sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:24:41.631838 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:41.631779 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b14adfd8-b845-4a2d-bf35-f8c45da4343f-maas-api-xks-ca" pod "b14adfd8-b845-4a2d-bf35-f8c45da4343f" (UID: "b14adfd8-b845-4a2d-bf35-f8c45da4343f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:24:41.731323 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:41.731291 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b14adfd8-b845-4a2d-bf35-f8c45da4343f-maas-api-xks-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:41.731323 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:41.731321 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr7sx\" (UniqueName: \"kubernetes.io/projected/b14adfd8-b845-4a2d-bf35-f8c45da4343f-kube-api-access-gr7sx\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:41.731539 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:41.731332 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b14adfd8-b845-4a2d-bf35-f8c45da4343f-maas-api-openshift-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:42.393816 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:42.393790 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b823784f-6jtzt" Nov 03 00:24:42.393995 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:42.393822 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b823784f-6jtzt" event={"ID":"b14adfd8-b845-4a2d-bf35-f8c45da4343f","Type":"ContainerDied","Data":"d6d418e812744a5f22a4111cc8d0b12ff3f699edec49f5772f291fa02f26aa6b"} Nov 03 00:24:42.393995 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:24:42.393874 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6d418e812744a5f22a4111cc8d0b12ff3f699edec49f5772f291fa02f26aa6b" Nov 03 00:24:57.845173 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:24:57.845120 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:25:27.845387 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:25:27.845346 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:25:30.045592 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:25:30.045553 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jh6w4"] Nov 03 00:25:30.094884 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:25:30.094851 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jh6w4"] Nov 03 00:25:31.882396 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:25:31.882361 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dda39dc-5fba-4455-b44a-4e2e248668ac" path="/var/lib/kubelet/pods/9dda39dc-5fba-4455-b44a-4e2e248668ac/volumes" Nov 03 00:25:57.852003 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:25:57.851957 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:26:05.047013 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:05.046978 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-l67sb"] Nov 03 00:26:05.095989 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:05.095955 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-l67sb"] Nov 03 00:26:05.882105 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:05.882070 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9db58498-2a5d-4971-9b2f-920a0a954395" path="/var/lib/kubelet/pods/9db58498-2a5d-4971-9b2f-920a0a954395/volumes" Nov 03 00:26:25.047019 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:25.046985 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tfwm9"] Nov 03 00:26:25.049411 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:25.049384 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tfwm9"] Nov 03 00:26:25.883214 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:25.883184 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21d05897-70cb-4441-abba-1e7136943153" path="/var/lib/kubelet/pods/21d05897-70cb-4441-abba-1e7136943153/volumes" Nov 03 00:26:27.845105 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:26:27.845066 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:26:36.103966 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:36.103927 5727 scope.go:122] "RemoveContainer" containerID="5ee034032f0a67fcd154e8bf670bc0eb4d18e966d0a01031901c19dc118672f0" Nov 03 00:26:37.046413 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:37.046380 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6l5zs"] Nov 03 00:26:37.096074 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:37.096046 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6l5zs"] Nov 03 00:26:37.881884 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:37.881842 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfd98fd6-35e5-4ae5-b591-a11f991b4b18" path="/var/lib/kubelet/pods/cfd98fd6-35e5-4ae5-b591-a11f991b4b18/volumes" Nov 03 00:26:57.305674 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:57.305638 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sj86z"] Nov 03 00:26:57.306153 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:57.306139 5727 cpu_manager.go:414] "containerMap: removing container" podUID="b14adfd8-b845-4a2d-bf35-f8c45da4343f" containerName="revoke-keys" Nov 03 00:26:57.306153 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:57.306153 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b14adfd8-b845-4a2d-bf35-f8c45da4343f" containerName="revoke-keys" Nov 03 00:26:57.306249 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:57.306233 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="b14adfd8-b845-4a2d-bf35-f8c45da4343f" containerName="revoke-keys" Nov 03 00:26:57.309348 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:57.309333 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sj86z" Nov 03 00:26:57.363544 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:57.363503 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:26:57.367109 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:57.367080 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sj86z"] Nov 03 00:26:57.412873 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:57.412840 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:26:57.413053 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:57.412931 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:26:57.413053 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:57.413026 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:26:57.423513 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:57.423462 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-579sg\" (UniqueName: \"kubernetes.io/projected/a9f5f6db-e1de-4901-b724-b95417c3ff6a-kube-api-access-579sg\") pod \"maas-api-revoke-sub-452b6d7b-sj86z\" (UID: \"a9f5f6db-e1de-4901-b724-b95417c3ff6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sj86z" Nov 03 00:26:57.423606 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:57.423533 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f5f6db-e1de-4901-b724-b95417c3ff6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-sj86z\" (UID: \"a9f5f6db-e1de-4901-b724-b95417c3ff6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sj86z" Nov 03 00:26:57.423680 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:57.423653 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f5f6db-e1de-4901-b724-b95417c3ff6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-sj86z\" (UID: \"a9f5f6db-e1de-4901-b724-b95417c3ff6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sj86z" Nov 03 00:26:57.524470 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:57.524437 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f5f6db-e1de-4901-b724-b95417c3ff6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-sj86z\" (UID: \"a9f5f6db-e1de-4901-b724-b95417c3ff6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sj86z" Nov 03 00:26:57.524639 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:57.524476 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f5f6db-e1de-4901-b724-b95417c3ff6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-sj86z\" (UID: \"a9f5f6db-e1de-4901-b724-b95417c3ff6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sj86z" Nov 03 00:26:57.524639 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:57.524529 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-579sg\" (UniqueName: \"kubernetes.io/projected/a9f5f6db-e1de-4901-b724-b95417c3ff6a-kube-api-access-579sg\") pod \"maas-api-revoke-sub-452b6d7b-sj86z\" (UID: \"a9f5f6db-e1de-4901-b724-b95417c3ff6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sj86z" Nov 03 00:26:57.525091 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:57.525070 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f5f6db-e1de-4901-b724-b95417c3ff6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-sj86z\" (UID: \"a9f5f6db-e1de-4901-b724-b95417c3ff6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sj86z" Nov 03 00:26:57.527049 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:57.527027 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f5f6db-e1de-4901-b724-b95417c3ff6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-sj86z\" (UID: \"a9f5f6db-e1de-4901-b724-b95417c3ff6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sj86z" Nov 03 00:26:57.587215 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:57.587189 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-579sg\" (UniqueName: \"kubernetes.io/projected/a9f5f6db-e1de-4901-b724-b95417c3ff6a-kube-api-access-579sg\") pod \"maas-api-revoke-sub-452b6d7b-sj86z\" (UID: \"a9f5f6db-e1de-4901-b724-b95417c3ff6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sj86z" Nov 03 00:26:57.622332 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:57.622304 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sj86z" Nov 03 00:26:57.847276 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:26:57.847234 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:26:58.001641 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:58.001614 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sj86z"] Nov 03 00:26:58.003171 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:26:58.003117 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9f5f6db_e1de_4901_b724_b95417c3ff6a.slice/crio-3e9ce49f252ac9cd32fa442f2e74a3a9b42259794b0f8f0ab4f2bc1eb0b4cb8c WatchSource:0}: Error finding container 3e9ce49f252ac9cd32fa442f2e74a3a9b42259794b0f8f0ab4f2bc1eb0b4cb8c: Status 404 returned error can't find the container with id 3e9ce49f252ac9cd32fa442f2e74a3a9b42259794b0f8f0ab4f2bc1eb0b4cb8c Nov 03 00:26:58.299369 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:58.299338 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sj86z" event={"ID":"a9f5f6db-e1de-4901-b724-b95417c3ff6a","Type":"ContainerStarted","Data":"d50fb81d0441768e8bfda3c9f2787be5cdc6d6d6ea849851f419067c18ee7103"} Nov 03 00:26:58.299545 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:58.299392 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sj86z" event={"ID":"a9f5f6db-e1de-4901-b724-b95417c3ff6a","Type":"ContainerStarted","Data":"3e9ce49f252ac9cd32fa442f2e74a3a9b42259794b0f8f0ab4f2bc1eb0b4cb8c"} Nov 03 00:26:58.414082 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:58.414005 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sj86z" podStartSLOduration=1.413987335 podStartE2EDuration="1.413987335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:26:58.41375742 +0000 UTC m=+5904.652255116" watchObservedRunningTime="2026-11-03 00:26:58.413987335 +0000 UTC m=+5904.652485001" Nov 03 00:26:59.311274 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:59.311172 5727 generic.go:356] "Generic (PLEG): container finished" podID="a9f5f6db-e1de-4901-b724-b95417c3ff6a" containerID="d50fb81d0441768e8bfda3c9f2787be5cdc6d6d6ea849851f419067c18ee7103" exitCode=0 Nov 03 00:26:59.311274 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:26:59.311267 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sj86z" event={"ID":"a9f5f6db-e1de-4901-b724-b95417c3ff6a","Type":"ContainerDied","Data":"d50fb81d0441768e8bfda3c9f2787be5cdc6d6d6ea849851f419067c18ee7103"} Nov 03 00:27:00.833168 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:00.832334 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sj86z" Nov 03 00:27:00.943762 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:00.943668 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9f5f6db-e1de-4901-b724-b95417c3ff6a-kube-api-access-579sg\" (UniqueName: \"kubernetes.io/projected/a9f5f6db-e1de-4901-b724-b95417c3ff6a-kube-api-access-579sg\") pod \"a9f5f6db-e1de-4901-b724-b95417c3ff6a\" (UID: \"a9f5f6db-e1de-4901-b724-b95417c3ff6a\") " Nov 03 00:27:00.944112 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:00.944091 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9f5f6db-e1de-4901-b724-b95417c3ff6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f5f6db-e1de-4901-b724-b95417c3ff6a-maas-api-xks-ca\") pod \"a9f5f6db-e1de-4901-b724-b95417c3ff6a\" (UID: \"a9f5f6db-e1de-4901-b724-b95417c3ff6a\") " Nov 03 00:27:00.944342 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:00.944324 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9f5f6db-e1de-4901-b724-b95417c3ff6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f5f6db-e1de-4901-b724-b95417c3ff6a-maas-api-openshift-ca\") pod \"a9f5f6db-e1de-4901-b724-b95417c3ff6a\" (UID: \"a9f5f6db-e1de-4901-b724-b95417c3ff6a\") " Nov 03 00:27:00.945277 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:00.945250 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9f5f6db-e1de-4901-b724-b95417c3ff6a-maas-api-openshift-ca" pod "a9f5f6db-e1de-4901-b724-b95417c3ff6a" (UID: "a9f5f6db-e1de-4901-b724-b95417c3ff6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:27:00.963221 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:00.949912 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9f5f6db-e1de-4901-b724-b95417c3ff6a-maas-api-xks-ca" pod "a9f5f6db-e1de-4901-b724-b95417c3ff6a" (UID: "a9f5f6db-e1de-4901-b724-b95417c3ff6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:27:00.970512 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:00.970253 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9f5f6db-e1de-4901-b724-b95417c3ff6a-kube-api-access-579sg" pod "a9f5f6db-e1de-4901-b724-b95417c3ff6a" (UID: "a9f5f6db-e1de-4901-b724-b95417c3ff6a"). InnerVolumeSpecName "kube-api-access-579sg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:27:01.046840 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:01.046792 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-579sg\" (UniqueName: \"kubernetes.io/projected/a9f5f6db-e1de-4901-b724-b95417c3ff6a-kube-api-access-579sg\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:27:01.047016 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:01.046845 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f5f6db-e1de-4901-b724-b95417c3ff6a-maas-api-xks-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:27:01.047016 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:01.046863 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f5f6db-e1de-4901-b724-b95417c3ff6a-maas-api-openshift-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:27:01.339233 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:01.339207 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sj86z" Nov 03 00:27:01.339413 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:01.339199 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sj86z" event={"ID":"a9f5f6db-e1de-4901-b724-b95417c3ff6a","Type":"ContainerDied","Data":"3e9ce49f252ac9cd32fa442f2e74a3a9b42259794b0f8f0ab4f2bc1eb0b4cb8c"} Nov 03 00:27:01.339413 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:01.339360 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e9ce49f252ac9cd32fa442f2e74a3a9b42259794b0f8f0ab4f2bc1eb0b4cb8c" Nov 03 00:27:27.846272 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:27:27.846232 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:27:30.867880 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:30.867847 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pnj52"] Nov 03 00:27:30.868652 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:30.868390 5727 cpu_manager.go:414] "containerMap: removing container" podUID="a9f5f6db-e1de-4901-b724-b95417c3ff6a" containerName="revoke-keys" Nov 03 00:27:30.868652 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:30.868403 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9f5f6db-e1de-4901-b724-b95417c3ff6a" containerName="revoke-keys" Nov 03 00:27:30.868652 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:30.868499 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9f5f6db-e1de-4901-b724-b95417c3ff6a" containerName="revoke-keys" Nov 03 00:27:30.871811 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:30.871786 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pnj52" Nov 03 00:27:30.925425 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:30.925382 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:27:30.925645 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:30.925497 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:27:30.974999 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:30.974964 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:27:30.975232 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:30.975057 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:27:30.977898 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:30.977868 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pnj52"] Nov 03 00:27:30.982623 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:30.982600 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/216fa6c9-c0de-4286-abbe-314fb439bb62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-pnj52\" (UID: \"216fa6c9-c0de-4286-abbe-314fb439bb62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pnj52" Nov 03 00:27:30.982768 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:30.982640 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpnsl\" (UniqueName: \"kubernetes.io/projected/216fa6c9-c0de-4286-abbe-314fb439bb62-kube-api-access-dpnsl\") pod \"maas-api-revoke-sub-3e31a506-pnj52\" (UID: \"216fa6c9-c0de-4286-abbe-314fb439bb62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pnj52" Nov 03 00:27:30.982768 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:30.982678 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/216fa6c9-c0de-4286-abbe-314fb439bb62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-pnj52\" (UID: \"216fa6c9-c0de-4286-abbe-314fb439bb62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pnj52" Nov 03 00:27:31.084166 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:31.084114 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/216fa6c9-c0de-4286-abbe-314fb439bb62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-pnj52\" (UID: \"216fa6c9-c0de-4286-abbe-314fb439bb62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pnj52" Nov 03 00:27:31.084362 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:31.084197 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpnsl\" (UniqueName: \"kubernetes.io/projected/216fa6c9-c0de-4286-abbe-314fb439bb62-kube-api-access-dpnsl\") pod \"maas-api-revoke-sub-3e31a506-pnj52\" (UID: \"216fa6c9-c0de-4286-abbe-314fb439bb62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pnj52" Nov 03 00:27:31.084362 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:31.084242 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/216fa6c9-c0de-4286-abbe-314fb439bb62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-pnj52\" (UID: \"216fa6c9-c0de-4286-abbe-314fb439bb62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pnj52" Nov 03 00:27:31.084927 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:31.084901 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/216fa6c9-c0de-4286-abbe-314fb439bb62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-pnj52\" (UID: \"216fa6c9-c0de-4286-abbe-314fb439bb62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pnj52" Nov 03 00:27:31.086901 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:31.086875 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/216fa6c9-c0de-4286-abbe-314fb439bb62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-pnj52\" (UID: \"216fa6c9-c0de-4286-abbe-314fb439bb62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pnj52" Nov 03 00:27:31.145211 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:31.145098 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpnsl\" (UniqueName: \"kubernetes.io/projected/216fa6c9-c0de-4286-abbe-314fb439bb62-kube-api-access-dpnsl\") pod \"maas-api-revoke-sub-3e31a506-pnj52\" (UID: \"216fa6c9-c0de-4286-abbe-314fb439bb62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pnj52" Nov 03 00:27:31.185991 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:31.185956 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pnj52" Nov 03 00:27:31.575222 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:31.570439 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pnj52"] Nov 03 00:27:31.575222 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:27:31.570671 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod216fa6c9_c0de_4286_abbe_314fb439bb62.slice/crio-11bc9ea47005ba2865daace3feefb48f74f21f00f01329d2aa9c2836b81d499d WatchSource:0}: Error finding container 11bc9ea47005ba2865daace3feefb48f74f21f00f01329d2aa9c2836b81d499d: Status 404 returned error can't find the container with id 11bc9ea47005ba2865daace3feefb48f74f21f00f01329d2aa9c2836b81d499d Nov 03 00:27:32.546401 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:32.546364 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pnj52" event={"ID":"216fa6c9-c0de-4286-abbe-314fb439bb62","Type":"ContainerStarted","Data":"25cfd900dceec3a7123262a1fd73ea1cd95416b443bb4201b3dd29bd99d4d937"} Nov 03 00:27:32.546820 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:32.546419 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pnj52" event={"ID":"216fa6c9-c0de-4286-abbe-314fb439bb62","Type":"ContainerStarted","Data":"11bc9ea47005ba2865daace3feefb48f74f21f00f01329d2aa9c2836b81d499d"} Nov 03 00:27:32.668875 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:32.667053 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pnj52" podStartSLOduration=2.667034188 podStartE2EDuration="2.667034188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:27:32.663578852 +0000 UTC m=+5938.902076538" watchObservedRunningTime="2026-11-03 00:27:32.667034188 +0000 UTC m=+5938.905531851" Nov 03 00:27:33.558558 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:33.558103 5727 generic.go:356] "Generic (PLEG): container finished" podID="216fa6c9-c0de-4286-abbe-314fb439bb62" containerID="25cfd900dceec3a7123262a1fd73ea1cd95416b443bb4201b3dd29bd99d4d937" exitCode=0 Nov 03 00:27:33.559240 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:33.558234 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pnj52" event={"ID":"216fa6c9-c0de-4286-abbe-314fb439bb62","Type":"ContainerDied","Data":"25cfd900dceec3a7123262a1fd73ea1cd95416b443bb4201b3dd29bd99d4d937"} Nov 03 00:27:34.718313 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:34.718287 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pnj52" Nov 03 00:27:34.849499 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:34.849416 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/216fa6c9-c0de-4286-abbe-314fb439bb62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/216fa6c9-c0de-4286-abbe-314fb439bb62-maas-api-openshift-ca\") pod \"216fa6c9-c0de-4286-abbe-314fb439bb62\" (UID: \"216fa6c9-c0de-4286-abbe-314fb439bb62\") " Nov 03 00:27:34.849499 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:34.849494 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/216fa6c9-c0de-4286-abbe-314fb439bb62-kube-api-access-dpnsl\" (UniqueName: \"kubernetes.io/projected/216fa6c9-c0de-4286-abbe-314fb439bb62-kube-api-access-dpnsl\") pod \"216fa6c9-c0de-4286-abbe-314fb439bb62\" (UID: \"216fa6c9-c0de-4286-abbe-314fb439bb62\") " Nov 03 00:27:34.849724 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:34.849656 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/216fa6c9-c0de-4286-abbe-314fb439bb62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/216fa6c9-c0de-4286-abbe-314fb439bb62-maas-api-xks-ca\") pod \"216fa6c9-c0de-4286-abbe-314fb439bb62\" (UID: \"216fa6c9-c0de-4286-abbe-314fb439bb62\") " Nov 03 00:27:34.849804 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:34.849780 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/216fa6c9-c0de-4286-abbe-314fb439bb62-maas-api-openshift-ca" pod "216fa6c9-c0de-4286-abbe-314fb439bb62" (UID: "216fa6c9-c0de-4286-abbe-314fb439bb62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:27:34.850052 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:34.850033 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/216fa6c9-c0de-4286-abbe-314fb439bb62-maas-api-openshift-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:27:34.851878 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:34.851846 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/216fa6c9-c0de-4286-abbe-314fb439bb62-kube-api-access-dpnsl" pod "216fa6c9-c0de-4286-abbe-314fb439bb62" (UID: "216fa6c9-c0de-4286-abbe-314fb439bb62"). InnerVolumeSpecName "kube-api-access-dpnsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:27:34.851984 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:34.851899 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/216fa6c9-c0de-4286-abbe-314fb439bb62-maas-api-xks-ca" pod "216fa6c9-c0de-4286-abbe-314fb439bb62" (UID: "216fa6c9-c0de-4286-abbe-314fb439bb62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:27:34.950647 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:34.950608 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpnsl\" (UniqueName: \"kubernetes.io/projected/216fa6c9-c0de-4286-abbe-314fb439bb62-kube-api-access-dpnsl\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:27:34.950647 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:34.950645 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/216fa6c9-c0de-4286-abbe-314fb439bb62-maas-api-xks-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:27:35.572772 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:35.572744 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pnj52" Nov 03 00:27:35.572953 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:35.572743 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pnj52" event={"ID":"216fa6c9-c0de-4286-abbe-314fb439bb62","Type":"ContainerDied","Data":"11bc9ea47005ba2865daace3feefb48f74f21f00f01329d2aa9c2836b81d499d"} Nov 03 00:27:35.572953 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:35.572900 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11bc9ea47005ba2865daace3feefb48f74f21f00f01329d2aa9c2836b81d499d" Nov 03 00:27:36.140061 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:36.140029 5727 scope.go:122] "RemoveContainer" containerID="4c6d81eb03519e30c124eab1f8578a79471ff56eaf4c342feb5e4a688cec9c22" Nov 03 00:27:36.150209 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:36.150187 5727 scope.go:122] "RemoveContainer" containerID="3dc1fe862db49f451de5da739d56280bd5ed31d52452c6882e106fc35c1dad32" Nov 03 00:27:36.159926 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:36.159906 5727 scope.go:122] "RemoveContainer" containerID="694f35a21e661c78ffd5e2ee142a028788c5fb1b334d7f4ba76264df0bf9c305" Nov 03 00:27:51.568319 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:51.568266 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-kvws4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:27:51.569007 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:51.568960 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" podUID="a5514e1f-1fb4-4158-85c8-f17fe2b94eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:27:56.182712 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:56.182669 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c1fb147-5jrzt"] Nov 03 00:27:56.183210 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:56.183193 5727 cpu_manager.go:414] "containerMap: removing container" podUID="216fa6c9-c0de-4286-abbe-314fb439bb62" containerName="revoke-keys" Nov 03 00:27:56.183265 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:56.183210 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="216fa6c9-c0de-4286-abbe-314fb439bb62" containerName="revoke-keys" Nov 03 00:27:56.183308 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:56.183293 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="216fa6c9-c0de-4286-abbe-314fb439bb62" containerName="revoke-keys" Nov 03 00:27:56.186393 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:56.186377 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c1fb147-5jrzt" Nov 03 00:27:56.243549 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:56.243513 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c1fb147-5jrzt"] Nov 03 00:27:56.288016 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:56.287977 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a034249-04ab-4783-9adb-d5ce03de6671-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c1fb147-5jrzt\" (UID: \"0a034249-04ab-4783-9adb-d5ce03de6671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c1fb147-5jrzt" Nov 03 00:27:56.288318 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:56.288110 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgc9n\" (UniqueName: \"kubernetes.io/projected/0a034249-04ab-4783-9adb-d5ce03de6671-kube-api-access-jgc9n\") pod \"maas-api-revoke-sub-7c1fb147-5jrzt\" (UID: \"0a034249-04ab-4783-9adb-d5ce03de6671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c1fb147-5jrzt" Nov 03 00:27:56.288318 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:56.288256 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a034249-04ab-4783-9adb-d5ce03de6671-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c1fb147-5jrzt\" (UID: \"0a034249-04ab-4783-9adb-d5ce03de6671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c1fb147-5jrzt" Nov 03 00:27:56.290858 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:56.290822 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:27:56.290858 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:56.290825 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:27:56.290858 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:56.290858 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:27:56.291163 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:56.290934 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:27:56.389025 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:56.388981 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgc9n\" (UniqueName: \"kubernetes.io/projected/0a034249-04ab-4783-9adb-d5ce03de6671-kube-api-access-jgc9n\") pod \"maas-api-revoke-sub-7c1fb147-5jrzt\" (UID: \"0a034249-04ab-4783-9adb-d5ce03de6671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c1fb147-5jrzt" Nov 03 00:27:56.389239 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:56.389069 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a034249-04ab-4783-9adb-d5ce03de6671-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c1fb147-5jrzt\" (UID: \"0a034249-04ab-4783-9adb-d5ce03de6671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c1fb147-5jrzt" Nov 03 00:27:56.389239 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:56.389172 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a034249-04ab-4783-9adb-d5ce03de6671-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c1fb147-5jrzt\" (UID: \"0a034249-04ab-4783-9adb-d5ce03de6671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c1fb147-5jrzt" Nov 03 00:27:56.389841 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:56.389813 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a034249-04ab-4783-9adb-d5ce03de6671-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c1fb147-5jrzt\" (UID: \"0a034249-04ab-4783-9adb-d5ce03de6671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c1fb147-5jrzt" Nov 03 00:27:56.391736 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:56.391705 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a034249-04ab-4783-9adb-d5ce03de6671-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c1fb147-5jrzt\" (UID: \"0a034249-04ab-4783-9adb-d5ce03de6671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c1fb147-5jrzt" Nov 03 00:27:56.450495 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:56.450424 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgc9n\" (UniqueName: \"kubernetes.io/projected/0a034249-04ab-4783-9adb-d5ce03de6671-kube-api-access-jgc9n\") pod \"maas-api-revoke-sub-7c1fb147-5jrzt\" (UID: \"0a034249-04ab-4783-9adb-d5ce03de6671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c1fb147-5jrzt" Nov 03 00:27:56.501391 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:56.501358 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c1fb147-5jrzt" Nov 03 00:27:56.879676 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:56.879648 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c1fb147-5jrzt"] Nov 03 00:27:56.881725 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:27:56.881694 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a034249_04ab_4783_9adb_d5ce03de6671.slice/crio-e39c225eb6cc6e90ad570e8f9b7d7b91208fca76288093154d3161e440473cb0 WatchSource:0}: Error finding container e39c225eb6cc6e90ad570e8f9b7d7b91208fca76288093154d3161e440473cb0: Status 404 returned error can't find the container with id e39c225eb6cc6e90ad570e8f9b7d7b91208fca76288093154d3161e440473cb0 Nov 03 00:27:57.740392 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:57.740358 5727 generic.go:356] "Generic (PLEG): container finished" podID="0a034249-04ab-4783-9adb-d5ce03de6671" containerID="db71d459959172e6d86b517a15eade9e533ee65958a7a42cf8a92475ebd14596" exitCode=0 Nov 03 00:27:57.740782 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:57.740453 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c1fb147-5jrzt" event={"ID":"0a034249-04ab-4783-9adb-d5ce03de6671","Type":"ContainerDied","Data":"db71d459959172e6d86b517a15eade9e533ee65958a7a42cf8a92475ebd14596"} Nov 03 00:27:57.740782 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:57.740531 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c1fb147-5jrzt" event={"ID":"0a034249-04ab-4783-9adb-d5ce03de6671","Type":"ContainerStarted","Data":"e39c225eb6cc6e90ad570e8f9b7d7b91208fca76288093154d3161e440473cb0"} Nov 03 00:27:57.847336 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:27:57.847231 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:27:58.889952 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:58.889930 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c1fb147-5jrzt" Nov 03 00:27:59.019786 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:59.019715 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a034249-04ab-4783-9adb-d5ce03de6671-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a034249-04ab-4783-9adb-d5ce03de6671-maas-api-xks-ca\") pod \"0a034249-04ab-4783-9adb-d5ce03de6671\" (UID: \"0a034249-04ab-4783-9adb-d5ce03de6671\") " Nov 03 00:27:59.019786 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:59.019782 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a034249-04ab-4783-9adb-d5ce03de6671-kube-api-access-jgc9n\" (UniqueName: \"kubernetes.io/projected/0a034249-04ab-4783-9adb-d5ce03de6671-kube-api-access-jgc9n\") pod \"0a034249-04ab-4783-9adb-d5ce03de6671\" (UID: \"0a034249-04ab-4783-9adb-d5ce03de6671\") " Nov 03 00:27:59.020027 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:59.019819 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a034249-04ab-4783-9adb-d5ce03de6671-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a034249-04ab-4783-9adb-d5ce03de6671-maas-api-openshift-ca\") pod \"0a034249-04ab-4783-9adb-d5ce03de6671\" (UID: \"0a034249-04ab-4783-9adb-d5ce03de6671\") " Nov 03 00:27:59.020282 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:59.020246 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a034249-04ab-4783-9adb-d5ce03de6671-maas-api-openshift-ca" pod "0a034249-04ab-4783-9adb-d5ce03de6671" (UID: "0a034249-04ab-4783-9adb-d5ce03de6671"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:27:59.022194 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:59.022165 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a034249-04ab-4783-9adb-d5ce03de6671-maas-api-xks-ca" pod "0a034249-04ab-4783-9adb-d5ce03de6671" (UID: "0a034249-04ab-4783-9adb-d5ce03de6671"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:27:59.022285 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:59.022263 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a034249-04ab-4783-9adb-d5ce03de6671-kube-api-access-jgc9n" pod "0a034249-04ab-4783-9adb-d5ce03de6671" (UID: "0a034249-04ab-4783-9adb-d5ce03de6671"). InnerVolumeSpecName "kube-api-access-jgc9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:27:59.120565 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:59.120537 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a034249-04ab-4783-9adb-d5ce03de6671-maas-api-xks-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:27:59.120565 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:59.120563 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgc9n\" (UniqueName: \"kubernetes.io/projected/0a034249-04ab-4783-9adb-d5ce03de6671-kube-api-access-jgc9n\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:27:59.120720 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:59.120573 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a034249-04ab-4783-9adb-d5ce03de6671-maas-api-openshift-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:27:59.754854 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:59.754820 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c1fb147-5jrzt" Nov 03 00:27:59.755068 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:59.754856 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c1fb147-5jrzt" event={"ID":"0a034249-04ab-4783-9adb-d5ce03de6671","Type":"ContainerDied","Data":"e39c225eb6cc6e90ad570e8f9b7d7b91208fca76288093154d3161e440473cb0"} Nov 03 00:27:59.755068 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:27:59.754903 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e39c225eb6cc6e90ad570e8f9b7d7b91208fca76288093154d3161e440473cb0" Nov 03 00:28:12.942402 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:12.942365 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-f9cfcc96b-mtll9"] Nov 03 00:28:12.943002 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:12.942962 5727 cpu_manager.go:414] "containerMap: removing container" podUID="0a034249-04ab-4783-9adb-d5ce03de6671" containerName="revoke-keys" Nov 03 00:28:12.943002 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:12.942977 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a034249-04ab-4783-9adb-d5ce03de6671" containerName="revoke-keys" Nov 03 00:28:12.943095 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:12.943048 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a034249-04ab-4783-9adb-d5ce03de6671" containerName="revoke-keys" Nov 03 00:28:12.946533 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:12.946518 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-f9cfcc96b-mtll9" Nov 03 00:28:13.003038 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:13.003009 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-f9cfcc96b-mtll9"] Nov 03 00:28:13.055529 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:13.055491 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj8dp\" (UniqueName: \"kubernetes.io/projected/13bfa8d3-0ac5-4912-abd6-ef22240f3bc4-kube-api-access-kj8dp\") pod \"payload-pre-processing-f9cfcc96b-mtll9\" (UID: \"13bfa8d3-0ac5-4912-abd6-ef22240f3bc4\") " pod="openshift-ingress/payload-pre-processing-f9cfcc96b-mtll9" Nov 03 00:28:13.055701 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:13.055603 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13bfa8d3-0ac5-4912-abd6-ef22240f3bc4-plugins-config-volume\") pod \"payload-pre-processing-f9cfcc96b-mtll9\" (UID: \"13bfa8d3-0ac5-4912-abd6-ef22240f3bc4\") " pod="openshift-ingress/payload-pre-processing-f9cfcc96b-mtll9" Nov 03 00:28:13.157113 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:13.157079 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj8dp\" (UniqueName: \"kubernetes.io/projected/13bfa8d3-0ac5-4912-abd6-ef22240f3bc4-kube-api-access-kj8dp\") pod \"payload-pre-processing-f9cfcc96b-mtll9\" (UID: \"13bfa8d3-0ac5-4912-abd6-ef22240f3bc4\") " pod="openshift-ingress/payload-pre-processing-f9cfcc96b-mtll9" Nov 03 00:28:13.157297 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:13.157161 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13bfa8d3-0ac5-4912-abd6-ef22240f3bc4-plugins-config-volume\") pod \"payload-pre-processing-f9cfcc96b-mtll9\" (UID: \"13bfa8d3-0ac5-4912-abd6-ef22240f3bc4\") " pod="openshift-ingress/payload-pre-processing-f9cfcc96b-mtll9" Nov 03 00:28:13.157846 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:13.157828 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13bfa8d3-0ac5-4912-abd6-ef22240f3bc4-plugins-config-volume\") pod \"payload-pre-processing-f9cfcc96b-mtll9\" (UID: \"13bfa8d3-0ac5-4912-abd6-ef22240f3bc4\") " pod="openshift-ingress/payload-pre-processing-f9cfcc96b-mtll9" Nov 03 00:28:13.218055 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:13.217984 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj8dp\" (UniqueName: \"kubernetes.io/projected/13bfa8d3-0ac5-4912-abd6-ef22240f3bc4-kube-api-access-kj8dp\") pod \"payload-pre-processing-f9cfcc96b-mtll9\" (UID: \"13bfa8d3-0ac5-4912-abd6-ef22240f3bc4\") " pod="openshift-ingress/payload-pre-processing-f9cfcc96b-mtll9" Nov 03 00:28:13.260200 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:13.260173 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-f9cfcc96b-mtll9" Nov 03 00:28:13.643017 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:13.642990 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-f9cfcc96b-mtll9"] Nov 03 00:28:13.644436 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:28:13.644410 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13bfa8d3_0ac5_4912_abd6_ef22240f3bc4.slice/crio-16d618cbf51a8bd89607cfd84157e98643e62167cdff1186be55ca73af38c2ba WatchSource:0}: Error finding container 16d618cbf51a8bd89607cfd84157e98643e62167cdff1186be55ca73af38c2ba: Status 404 returned error can't find the container with id 16d618cbf51a8bd89607cfd84157e98643e62167cdff1186be55ca73af38c2ba Nov 03 00:28:13.844371 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:13.844332 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-f9cfcc96b-mtll9" event={"ID":"13bfa8d3-0ac5-4912-abd6-ef22240f3bc4","Type":"ContainerStarted","Data":"b16e485c44b561ce9f5aa4335d6e41aaea44d171581c56a13a81a445ad5b1e55"} Nov 03 00:28:13.844528 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:13.844385 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-f9cfcc96b-mtll9" event={"ID":"13bfa8d3-0ac5-4912-abd6-ef22240f3bc4","Type":"ContainerStarted","Data":"16d618cbf51a8bd89607cfd84157e98643e62167cdff1186be55ca73af38c2ba"} Nov 03 00:28:13.844528 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:13.844520 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-f9cfcc96b-mtll9" Nov 03 00:28:13.960088 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:13.959970 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-f9cfcc96b-mtll9" podStartSLOduration=1.959955391 podStartE2EDuration="1.959955391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:28:13.958663074 +0000 UTC m=+5980.197160786" watchObservedRunningTime="2026-11-03 00:28:13.959955391 +0000 UTC m=+5980.198453053" Nov 03 00:28:24.854031 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:24.854002 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-f9cfcc96b-mtll9" Nov 03 00:28:25.024577 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:25.024534 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-cdflj"] Nov 03 00:28:27.007974 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:27.007944 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-78846c8684-4ns6f"] Nov 03 00:28:27.011819 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:27.011803 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-78846c8684-4ns6f" Nov 03 00:28:27.069593 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:27.069552 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-78846c8684-4ns6f"] Nov 03 00:28:27.084717 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:27.084684 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbg99\" (UniqueName: \"kubernetes.io/projected/039c7313-ed6a-400b-85bb-ca3d4352c445-kube-api-access-fbg99\") pod \"payload-processing-78846c8684-4ns6f\" (UID: \"039c7313-ed6a-400b-85bb-ca3d4352c445\") " pod="openshift-ingress/payload-processing-78846c8684-4ns6f" Nov 03 00:28:27.084849 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:27.084812 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/039c7313-ed6a-400b-85bb-ca3d4352c445-plugins-config-volume\") pod \"payload-processing-78846c8684-4ns6f\" (UID: \"039c7313-ed6a-400b-85bb-ca3d4352c445\") " pod="openshift-ingress/payload-processing-78846c8684-4ns6f" Nov 03 00:28:27.186113 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:27.186082 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbg99\" (UniqueName: \"kubernetes.io/projected/039c7313-ed6a-400b-85bb-ca3d4352c445-kube-api-access-fbg99\") pod \"payload-processing-78846c8684-4ns6f\" (UID: \"039c7313-ed6a-400b-85bb-ca3d4352c445\") " pod="openshift-ingress/payload-processing-78846c8684-4ns6f" Nov 03 00:28:27.186326 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:27.186165 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/039c7313-ed6a-400b-85bb-ca3d4352c445-plugins-config-volume\") pod \"payload-processing-78846c8684-4ns6f\" (UID: \"039c7313-ed6a-400b-85bb-ca3d4352c445\") " pod="openshift-ingress/payload-processing-78846c8684-4ns6f" Nov 03 00:28:27.186903 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:27.186875 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/039c7313-ed6a-400b-85bb-ca3d4352c445-plugins-config-volume\") pod \"payload-processing-78846c8684-4ns6f\" (UID: \"039c7313-ed6a-400b-85bb-ca3d4352c445\") " pod="openshift-ingress/payload-processing-78846c8684-4ns6f" Nov 03 00:28:27.247170 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:27.247120 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbg99\" (UniqueName: \"kubernetes.io/projected/039c7313-ed6a-400b-85bb-ca3d4352c445-kube-api-access-fbg99\") pod \"payload-processing-78846c8684-4ns6f\" (UID: \"039c7313-ed6a-400b-85bb-ca3d4352c445\") " pod="openshift-ingress/payload-processing-78846c8684-4ns6f" Nov 03 00:28:27.325218 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:27.325189 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-78846c8684-4ns6f" Nov 03 00:28:27.699689 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:27.699659 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-78846c8684-4ns6f"] Nov 03 00:28:27.701812 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:28:27.701784 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod039c7313_ed6a_400b_85bb_ca3d4352c445.slice/crio-579d2403c16f2d59ae5f1aad5cb0822819757574876c10a68160104abc179f05 WatchSource:0}: Error finding container 579d2403c16f2d59ae5f1aad5cb0822819757574876c10a68160104abc179f05: Status 404 returned error can't find the container with id 579d2403c16f2d59ae5f1aad5cb0822819757574876c10a68160104abc179f05 Nov 03 00:28:27.843809 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:28:27.843763 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:28:27.931538 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:27.931495 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-78846c8684-4ns6f" event={"ID":"039c7313-ed6a-400b-85bb-ca3d4352c445","Type":"ContainerStarted","Data":"073735c8e8b3faf2b8b608484a4562d5060af4b767dce598713f130d15b0ef75"} Nov 03 00:28:27.931743 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:27.931556 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-78846c8684-4ns6f" event={"ID":"039c7313-ed6a-400b-85bb-ca3d4352c445","Type":"ContainerStarted","Data":"579d2403c16f2d59ae5f1aad5cb0822819757574876c10a68160104abc179f05"} Nov 03 00:28:27.931743 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:27.931583 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-78846c8684-4ns6f" Nov 03 00:28:28.047926 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:28.047827 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-78846c8684-4ns6f" podStartSLOduration=2.047813475 podStartE2EDuration="2.047813475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:28:28.046464823 +0000 UTC m=+5994.284962500" watchObservedRunningTime="2026-11-03 00:28:28.047813475 +0000 UTC m=+5994.286311137" Nov 03 00:28:30.025657 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:30.025606 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cdflj" podUID="c0c34a06-6f76-4573-9cc8-7ba13cc79280" containerName="payload-pre-processing" containerID="cri-o://0739013299d944ad22f4ad12b21e669e27573314fce73b181ec8db3eafb6f853" gracePeriod=25 Nov 03 00:28:30.288442 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:30.288383 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cdflj" Nov 03 00:28:30.319692 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:30.319668 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0c34a06-6f76-4573-9cc8-7ba13cc79280-kube-api-access-mjptc\" (UniqueName: \"kubernetes.io/projected/c0c34a06-6f76-4573-9cc8-7ba13cc79280-kube-api-access-mjptc\") pod \"c0c34a06-6f76-4573-9cc8-7ba13cc79280\" (UID: \"c0c34a06-6f76-4573-9cc8-7ba13cc79280\") " Nov 03 00:28:30.319814 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:30.319712 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0c34a06-6f76-4573-9cc8-7ba13cc79280-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0c34a06-6f76-4573-9cc8-7ba13cc79280-plugins-config-volume\") pod \"c0c34a06-6f76-4573-9cc8-7ba13cc79280\" (UID: \"c0c34a06-6f76-4573-9cc8-7ba13cc79280\") " Nov 03 00:28:30.320145 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:30.320102 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0c34a06-6f76-4573-9cc8-7ba13cc79280-plugins-config-volume" pod "c0c34a06-6f76-4573-9cc8-7ba13cc79280" (UID: "c0c34a06-6f76-4573-9cc8-7ba13cc79280"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:28:30.321941 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:30.321914 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c34a06-6f76-4573-9cc8-7ba13cc79280-kube-api-access-mjptc" pod "c0c34a06-6f76-4573-9cc8-7ba13cc79280" (UID: "c0c34a06-6f76-4573-9cc8-7ba13cc79280"). InnerVolumeSpecName "kube-api-access-mjptc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:28:30.421161 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:30.421110 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjptc\" (UniqueName: \"kubernetes.io/projected/c0c34a06-6f76-4573-9cc8-7ba13cc79280-kube-api-access-mjptc\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:28:30.421161 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:30.421160 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0c34a06-6f76-4573-9cc8-7ba13cc79280-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:28:30.953589 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:30.953558 5727 generic.go:356] "Generic (PLEG): container finished" podID="c0c34a06-6f76-4573-9cc8-7ba13cc79280" containerID="0739013299d944ad22f4ad12b21e669e27573314fce73b181ec8db3eafb6f853" exitCode=0 Nov 03 00:28:30.953772 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:30.953603 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cdflj" Nov 03 00:28:30.953772 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:30.953629 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cdflj" event={"ID":"c0c34a06-6f76-4573-9cc8-7ba13cc79280","Type":"ContainerDied","Data":"0739013299d944ad22f4ad12b21e669e27573314fce73b181ec8db3eafb6f853"} Nov 03 00:28:30.953772 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:30.953668 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cdflj" event={"ID":"c0c34a06-6f76-4573-9cc8-7ba13cc79280","Type":"ContainerDied","Data":"46950a0261aef0bb4e0e10868f31e3946b2753a9db89b025aae343c47bfee4f5"} Nov 03 00:28:30.953772 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:30.953687 5727 scope.go:122] "RemoveContainer" containerID="0739013299d944ad22f4ad12b21e669e27573314fce73b181ec8db3eafb6f853" Nov 03 00:28:30.963654 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:30.963635 5727 scope.go:122] "RemoveContainer" containerID="0739013299d944ad22f4ad12b21e669e27573314fce73b181ec8db3eafb6f853" Nov 03 00:28:30.963911 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:28:30.963891 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0739013299d944ad22f4ad12b21e669e27573314fce73b181ec8db3eafb6f853\": container with ID starting with 0739013299d944ad22f4ad12b21e669e27573314fce73b181ec8db3eafb6f853 not found: ID does not exist" containerID="0739013299d944ad22f4ad12b21e669e27573314fce73b181ec8db3eafb6f853" Nov 03 00:28:30.963989 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:30.963918 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0739013299d944ad22f4ad12b21e669e27573314fce73b181ec8db3eafb6f853"} err="failed to get container status \"0739013299d944ad22f4ad12b21e669e27573314fce73b181ec8db3eafb6f853\": rpc error: code = NotFound desc = could not find container \"0739013299d944ad22f4ad12b21e669e27573314fce73b181ec8db3eafb6f853\": container with ID starting with 0739013299d944ad22f4ad12b21e669e27573314fce73b181ec8db3eafb6f853 not found: ID does not exist" Nov 03 00:28:31.122178 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:31.122144 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-cdflj"] Nov 03 00:28:31.125329 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:31.125301 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-cdflj"] Nov 03 00:28:31.882540 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:31.882508 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0c34a06-6f76-4573-9cc8-7ba13cc79280" path="/var/lib/kubelet/pods/c0c34a06-6f76-4573-9cc8-7ba13cc79280/volumes" Nov 03 00:28:35.164892 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:35.164864 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 03 00:28:35.181066 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:35.181040 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 03 00:28:35.275111 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:35.275087 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 03 00:28:35.289729 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:35.289706 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 03 00:28:38.940967 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:38.940933 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-78846c8684-4ns6f" Nov 03 00:28:39.108435 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:39.108400 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-279zt"] Nov 03 00:28:41.205335 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:41.205302 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc7f58bb9-9dh6m"] Nov 03 00:28:41.205818 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:41.205805 5727 cpu_manager.go:414] "containerMap: removing container" podUID="c0c34a06-6f76-4573-9cc8-7ba13cc79280" containerName="payload-pre-processing" Nov 03 00:28:41.205864 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:41.205819 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0c34a06-6f76-4573-9cc8-7ba13cc79280" containerName="payload-pre-processing" Nov 03 00:28:41.205905 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:41.205900 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0c34a06-6f76-4573-9cc8-7ba13cc79280" containerName="payload-pre-processing" Nov 03 00:28:41.210646 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:41.210629 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc7f58bb9-9dh6m" Nov 03 00:28:41.266233 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:41.266203 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc7f58bb9-9dh6m"] Nov 03 00:28:41.328465 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:41.328431 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6ba592a-8ce9-44bd-9ade-8dfd408bad2a-plugins-config-volume\") pod \"payload-processing-5cc7f58bb9-9dh6m\" (UID: \"f6ba592a-8ce9-44bd-9ade-8dfd408bad2a\") " pod="openshift-ingress/payload-processing-5cc7f58bb9-9dh6m" Nov 03 00:28:41.328609 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:41.328570 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpl8s\" (UniqueName: \"kubernetes.io/projected/f6ba592a-8ce9-44bd-9ade-8dfd408bad2a-kube-api-access-lpl8s\") pod \"payload-processing-5cc7f58bb9-9dh6m\" (UID: \"f6ba592a-8ce9-44bd-9ade-8dfd408bad2a\") " pod="openshift-ingress/payload-processing-5cc7f58bb9-9dh6m" Nov 03 00:28:41.363506 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:41.363475 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-75bc6697f4-778p5"] Nov 03 00:28:41.373578 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:41.373545 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-75bc6697f4-778p5" Nov 03 00:28:41.424717 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:41.424681 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-75bc6697f4-778p5"] Nov 03 00:28:41.429968 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:41.429941 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5nx4\" (UniqueName: \"kubernetes.io/projected/36f33895-59f7-4533-9478-fcd8299a8b4f-kube-api-access-g5nx4\") pod \"payload-pre-processing-75bc6697f4-778p5\" (UID: \"36f33895-59f7-4533-9478-fcd8299a8b4f\") " pod="openshift-ingress/payload-pre-processing-75bc6697f4-778p5" Nov 03 00:28:41.430110 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:41.429986 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36f33895-59f7-4533-9478-fcd8299a8b4f-plugins-config-volume\") pod \"payload-pre-processing-75bc6697f4-778p5\" (UID: \"36f33895-59f7-4533-9478-fcd8299a8b4f\") " pod="openshift-ingress/payload-pre-processing-75bc6697f4-778p5" Nov 03 00:28:41.430110 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:41.430018 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6ba592a-8ce9-44bd-9ade-8dfd408bad2a-plugins-config-volume\") pod \"payload-processing-5cc7f58bb9-9dh6m\" (UID: \"f6ba592a-8ce9-44bd-9ade-8dfd408bad2a\") " pod="openshift-ingress/payload-processing-5cc7f58bb9-9dh6m" Nov 03 00:28:41.430269 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:41.430252 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpl8s\" (UniqueName: \"kubernetes.io/projected/f6ba592a-8ce9-44bd-9ade-8dfd408bad2a-kube-api-access-lpl8s\") pod \"payload-processing-5cc7f58bb9-9dh6m\" (UID: \"f6ba592a-8ce9-44bd-9ade-8dfd408bad2a\") " pod="openshift-ingress/payload-processing-5cc7f58bb9-9dh6m" Nov 03 00:28:41.430851 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:41.430832 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6ba592a-8ce9-44bd-9ade-8dfd408bad2a-plugins-config-volume\") pod \"payload-processing-5cc7f58bb9-9dh6m\" (UID: \"f6ba592a-8ce9-44bd-9ade-8dfd408bad2a\") " pod="openshift-ingress/payload-processing-5cc7f58bb9-9dh6m" Nov 03 00:28:41.493750 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:41.493680 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpl8s\" (UniqueName: \"kubernetes.io/projected/f6ba592a-8ce9-44bd-9ade-8dfd408bad2a-kube-api-access-lpl8s\") pod \"payload-processing-5cc7f58bb9-9dh6m\" (UID: \"f6ba592a-8ce9-44bd-9ade-8dfd408bad2a\") " pod="openshift-ingress/payload-processing-5cc7f58bb9-9dh6m" Nov 03 00:28:41.524027 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:41.523997 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc7f58bb9-9dh6m" Nov 03 00:28:41.531515 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:41.531486 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36f33895-59f7-4533-9478-fcd8299a8b4f-plugins-config-volume\") pod \"payload-pre-processing-75bc6697f4-778p5\" (UID: \"36f33895-59f7-4533-9478-fcd8299a8b4f\") " pod="openshift-ingress/payload-pre-processing-75bc6697f4-778p5" Nov 03 00:28:41.531681 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:41.531667 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5nx4\" (UniqueName: \"kubernetes.io/projected/36f33895-59f7-4533-9478-fcd8299a8b4f-kube-api-access-g5nx4\") pod \"payload-pre-processing-75bc6697f4-778p5\" (UID: \"36f33895-59f7-4533-9478-fcd8299a8b4f\") " pod="openshift-ingress/payload-pre-processing-75bc6697f4-778p5" Nov 03 00:28:41.532142 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:41.532111 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36f33895-59f7-4533-9478-fcd8299a8b4f-plugins-config-volume\") pod \"payload-pre-processing-75bc6697f4-778p5\" (UID: \"36f33895-59f7-4533-9478-fcd8299a8b4f\") " pod="openshift-ingress/payload-pre-processing-75bc6697f4-778p5" Nov 03 00:28:41.595083 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:41.595021 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5nx4\" (UniqueName: \"kubernetes.io/projected/36f33895-59f7-4533-9478-fcd8299a8b4f-kube-api-access-g5nx4\") pod \"payload-pre-processing-75bc6697f4-778p5\" (UID: \"36f33895-59f7-4533-9478-fcd8299a8b4f\") " pod="openshift-ingress/payload-pre-processing-75bc6697f4-778p5" Nov 03 00:28:41.687974 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:41.687947 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-75bc6697f4-778p5" Nov 03 00:28:41.898763 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:41.898736 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc7f58bb9-9dh6m"] Nov 03 00:28:41.900634 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:28:41.900600 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ba592a_8ce9_44bd_9ade_8dfd408bad2a.slice/crio-4ba58560837fee552bb8620716b118c608177164cbb1088159389eac9ccc49d6 WatchSource:0}: Error finding container 4ba58560837fee552bb8620716b118c608177164cbb1088159389eac9ccc49d6: Status 404 returned error can't find the container with id 4ba58560837fee552bb8620716b118c608177164cbb1088159389eac9ccc49d6 Nov 03 00:28:42.023778 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:42.023701 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc7f58bb9-9dh6m" event={"ID":"f6ba592a-8ce9-44bd-9ade-8dfd408bad2a","Type":"ContainerStarted","Data":"46197dbe07a28b502ac708651466ff6962e2f0911c2a6dded6b9c780593814f2"} Nov 03 00:28:42.023778 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:42.023764 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc7f58bb9-9dh6m" event={"ID":"f6ba592a-8ce9-44bd-9ade-8dfd408bad2a","Type":"ContainerStarted","Data":"4ba58560837fee552bb8620716b118c608177164cbb1088159389eac9ccc49d6"} Nov 03 00:28:42.023988 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:42.023911 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc7f58bb9-9dh6m" Nov 03 00:28:42.062775 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:42.062750 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-75bc6697f4-778p5"] Nov 03 00:28:42.064083 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:28:42.064055 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f33895_59f7_4533_9478_fcd8299a8b4f.slice/crio-23b95ce84545f6777951f066dd92fd21ba4876ee195cf9a24df0917188c0f301 WatchSource:0}: Error finding container 23b95ce84545f6777951f066dd92fd21ba4876ee195cf9a24df0917188c0f301: Status 404 returned error can't find the container with id 23b95ce84545f6777951f066dd92fd21ba4876ee195cf9a24df0917188c0f301 Nov 03 00:28:42.141893 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:42.141849 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc7f58bb9-9dh6m" podStartSLOduration=1.141834734 podStartE2EDuration="1.141834734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:28:42.139451404 +0000 UTC m=+6008.377949077" watchObservedRunningTime="2026-11-03 00:28:42.141834734 +0000 UTC m=+6008.380332395" Nov 03 00:28:43.032548 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:43.032516 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-75bc6697f4-778p5" event={"ID":"36f33895-59f7-4533-9478-fcd8299a8b4f","Type":"ContainerStarted","Data":"0014ac2e0834db6306adb2369ad734b9cb1eabec3a6b3e66402b9937c1cab276"} Nov 03 00:28:43.032947 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:43.032568 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-75bc6697f4-778p5" event={"ID":"36f33895-59f7-4533-9478-fcd8299a8b4f","Type":"ContainerStarted","Data":"23b95ce84545f6777951f066dd92fd21ba4876ee195cf9a24df0917188c0f301"} Nov 03 00:28:43.032947 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:43.032607 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-75bc6697f4-778p5" Nov 03 00:28:43.148720 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:43.148672 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-75bc6697f4-778p5" podStartSLOduration=2.148654337 podStartE2EDuration="2.148654337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:28:43.147407104 +0000 UTC m=+6009.385904789" watchObservedRunningTime="2026-11-03 00:28:43.148654337 +0000 UTC m=+6009.387151998" Nov 03 00:28:44.109660 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:44.109621 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-279zt" podUID="f36665e4-38c7-4d93-b047-306185b1a17f" containerName="payload-processing" containerID="cri-o://76b3e32e5099d40d1029f75312d4017749e2d499b1e86121d04fcffa9cc12973" gracePeriod=25 Nov 03 00:28:44.371565 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:44.371504 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-279zt" Nov 03 00:28:44.460183 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:44.460123 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f36665e4-38c7-4d93-b047-306185b1a17f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f36665e4-38c7-4d93-b047-306185b1a17f-plugins-config-volume\") pod \"f36665e4-38c7-4d93-b047-306185b1a17f\" (UID: \"f36665e4-38c7-4d93-b047-306185b1a17f\") " Nov 03 00:28:44.460359 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:44.460265 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f36665e4-38c7-4d93-b047-306185b1a17f-kube-api-access-9qvmf\" (UniqueName: \"kubernetes.io/projected/f36665e4-38c7-4d93-b047-306185b1a17f-kube-api-access-9qvmf\") pod \"f36665e4-38c7-4d93-b047-306185b1a17f\" (UID: \"f36665e4-38c7-4d93-b047-306185b1a17f\") " Nov 03 00:28:44.460486 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:44.460464 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f36665e4-38c7-4d93-b047-306185b1a17f-plugins-config-volume" pod "f36665e4-38c7-4d93-b047-306185b1a17f" (UID: "f36665e4-38c7-4d93-b047-306185b1a17f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:28:44.460598 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:44.460584 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f36665e4-38c7-4d93-b047-306185b1a17f-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:28:44.462728 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:44.462697 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f36665e4-38c7-4d93-b047-306185b1a17f-kube-api-access-9qvmf" pod "f36665e4-38c7-4d93-b047-306185b1a17f" (UID: "f36665e4-38c7-4d93-b047-306185b1a17f"). InnerVolumeSpecName "kube-api-access-9qvmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:28:44.561223 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:44.561195 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qvmf\" (UniqueName: \"kubernetes.io/projected/f36665e4-38c7-4d93-b047-306185b1a17f-kube-api-access-9qvmf\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:28:45.045563 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:45.045529 5727 generic.go:356] "Generic (PLEG): container finished" podID="f36665e4-38c7-4d93-b047-306185b1a17f" containerID="76b3e32e5099d40d1029f75312d4017749e2d499b1e86121d04fcffa9cc12973" exitCode=0 Nov 03 00:28:45.045745 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:45.045573 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-279zt" Nov 03 00:28:45.045745 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:45.045610 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-279zt" event={"ID":"f36665e4-38c7-4d93-b047-306185b1a17f","Type":"ContainerDied","Data":"76b3e32e5099d40d1029f75312d4017749e2d499b1e86121d04fcffa9cc12973"} Nov 03 00:28:45.045745 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:45.045665 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-279zt" event={"ID":"f36665e4-38c7-4d93-b047-306185b1a17f","Type":"ContainerDied","Data":"8026f4378626f6290c45434097f10ad504476324e776d23e08d6e4104f4374cd"} Nov 03 00:28:45.045745 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:45.045686 5727 scope.go:122] "RemoveContainer" containerID="76b3e32e5099d40d1029f75312d4017749e2d499b1e86121d04fcffa9cc12973" Nov 03 00:28:45.058183 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:45.058166 5727 scope.go:122] "RemoveContainer" containerID="76b3e32e5099d40d1029f75312d4017749e2d499b1e86121d04fcffa9cc12973" Nov 03 00:28:45.058459 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:28:45.058441 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76b3e32e5099d40d1029f75312d4017749e2d499b1e86121d04fcffa9cc12973\": container with ID starting with 76b3e32e5099d40d1029f75312d4017749e2d499b1e86121d04fcffa9cc12973 not found: ID does not exist" containerID="76b3e32e5099d40d1029f75312d4017749e2d499b1e86121d04fcffa9cc12973" Nov 03 00:28:45.058530 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:45.058466 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76b3e32e5099d40d1029f75312d4017749e2d499b1e86121d04fcffa9cc12973"} err="failed to get container status \"76b3e32e5099d40d1029f75312d4017749e2d499b1e86121d04fcffa9cc12973\": rpc error: code = NotFound desc = could not find container \"76b3e32e5099d40d1029f75312d4017749e2d499b1e86121d04fcffa9cc12973\": container with ID starting with 76b3e32e5099d40d1029f75312d4017749e2d499b1e86121d04fcffa9cc12973 not found: ID does not exist" Nov 03 00:28:45.210319 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:45.210288 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-279zt"] Nov 03 00:28:45.214527 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:45.214501 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-279zt"] Nov 03 00:28:45.882119 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:45.882088 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f36665e4-38c7-4d93-b047-306185b1a17f" path="/var/lib/kubelet/pods/f36665e4-38c7-4d93-b047-306185b1a17f/volumes" Nov 03 00:28:53.035047 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:53.035016 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc7f58bb9-9dh6m" Nov 03 00:28:53.181078 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:53.181046 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-78846c8684-4ns6f"] Nov 03 00:28:54.041703 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:54.041669 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-75bc6697f4-778p5" Nov 03 00:28:54.210704 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:54.210673 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-f9cfcc96b-mtll9"] Nov 03 00:28:57.845610 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:28:57.845571 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:28:58.181952 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:58.181866 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-78846c8684-4ns6f" podUID="039c7313-ed6a-400b-85bb-ca3d4352c445" containerName="payload-processing" containerID="cri-o://073735c8e8b3faf2b8b608484a4562d5060af4b767dce598713f130d15b0ef75" gracePeriod=25 Nov 03 00:28:58.449388 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:58.449335 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-78846c8684-4ns6f" Nov 03 00:28:58.596159 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:58.596088 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/039c7313-ed6a-400b-85bb-ca3d4352c445-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/039c7313-ed6a-400b-85bb-ca3d4352c445-plugins-config-volume\") pod \"039c7313-ed6a-400b-85bb-ca3d4352c445\" (UID: \"039c7313-ed6a-400b-85bb-ca3d4352c445\") " Nov 03 00:28:58.596362 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:58.596286 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/039c7313-ed6a-400b-85bb-ca3d4352c445-kube-api-access-fbg99\" (UniqueName: \"kubernetes.io/projected/039c7313-ed6a-400b-85bb-ca3d4352c445-kube-api-access-fbg99\") pod \"039c7313-ed6a-400b-85bb-ca3d4352c445\" (UID: \"039c7313-ed6a-400b-85bb-ca3d4352c445\") " Nov 03 00:28:58.596568 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:58.596533 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/039c7313-ed6a-400b-85bb-ca3d4352c445-plugins-config-volume" pod "039c7313-ed6a-400b-85bb-ca3d4352c445" (UID: "039c7313-ed6a-400b-85bb-ca3d4352c445"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:28:58.596740 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:58.596721 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/039c7313-ed6a-400b-85bb-ca3d4352c445-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:28:58.598812 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:58.598780 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/039c7313-ed6a-400b-85bb-ca3d4352c445-kube-api-access-fbg99" pod "039c7313-ed6a-400b-85bb-ca3d4352c445" (UID: "039c7313-ed6a-400b-85bb-ca3d4352c445"). InnerVolumeSpecName "kube-api-access-fbg99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:28:58.697587 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:58.697558 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbg99\" (UniqueName: \"kubernetes.io/projected/039c7313-ed6a-400b-85bb-ca3d4352c445-kube-api-access-fbg99\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:28:59.135371 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:59.135339 5727 generic.go:356] "Generic (PLEG): container finished" podID="039c7313-ed6a-400b-85bb-ca3d4352c445" containerID="073735c8e8b3faf2b8b608484a4562d5060af4b767dce598713f130d15b0ef75" exitCode=0 Nov 03 00:28:59.135818 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:59.135384 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-78846c8684-4ns6f" Nov 03 00:28:59.135818 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:59.135430 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-78846c8684-4ns6f" event={"ID":"039c7313-ed6a-400b-85bb-ca3d4352c445","Type":"ContainerDied","Data":"073735c8e8b3faf2b8b608484a4562d5060af4b767dce598713f130d15b0ef75"} Nov 03 00:28:59.135818 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:59.135494 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-78846c8684-4ns6f" event={"ID":"039c7313-ed6a-400b-85bb-ca3d4352c445","Type":"ContainerDied","Data":"579d2403c16f2d59ae5f1aad5cb0822819757574876c10a68160104abc179f05"} Nov 03 00:28:59.135818 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:59.135521 5727 scope.go:122] "RemoveContainer" containerID="073735c8e8b3faf2b8b608484a4562d5060af4b767dce598713f130d15b0ef75" Nov 03 00:28:59.145935 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:59.145916 5727 scope.go:122] "RemoveContainer" containerID="073735c8e8b3faf2b8b608484a4562d5060af4b767dce598713f130d15b0ef75" Nov 03 00:28:59.146279 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:28:59.146252 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"073735c8e8b3faf2b8b608484a4562d5060af4b767dce598713f130d15b0ef75\": container with ID starting with 073735c8e8b3faf2b8b608484a4562d5060af4b767dce598713f130d15b0ef75 not found: ID does not exist" containerID="073735c8e8b3faf2b8b608484a4562d5060af4b767dce598713f130d15b0ef75" Nov 03 00:28:59.146340 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:59.146289 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"073735c8e8b3faf2b8b608484a4562d5060af4b767dce598713f130d15b0ef75"} err="failed to get container status \"073735c8e8b3faf2b8b608484a4562d5060af4b767dce598713f130d15b0ef75\": rpc error: code = NotFound desc = could not find container \"073735c8e8b3faf2b8b608484a4562d5060af4b767dce598713f130d15b0ef75\": container with ID starting with 073735c8e8b3faf2b8b608484a4562d5060af4b767dce598713f130d15b0ef75 not found: ID does not exist" Nov 03 00:28:59.211138 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:59.211103 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-f9cfcc96b-mtll9" podUID="13bfa8d3-0ac5-4912-abd6-ef22240f3bc4" containerName="payload-pre-processing" containerID="cri-o://b16e485c44b561ce9f5aa4335d6e41aaea44d171581c56a13a81a445ad5b1e55" gracePeriod=25 Nov 03 00:28:59.299210 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:59.299183 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-78846c8684-4ns6f"] Nov 03 00:28:59.303068 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:59.303046 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-78846c8684-4ns6f"] Nov 03 00:28:59.461732 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:59.461710 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-f9cfcc96b-mtll9" Nov 03 00:28:59.606117 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:59.606087 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13bfa8d3-0ac5-4912-abd6-ef22240f3bc4-kube-api-access-kj8dp\" (UniqueName: \"kubernetes.io/projected/13bfa8d3-0ac5-4912-abd6-ef22240f3bc4-kube-api-access-kj8dp\") pod \"13bfa8d3-0ac5-4912-abd6-ef22240f3bc4\" (UID: \"13bfa8d3-0ac5-4912-abd6-ef22240f3bc4\") " Nov 03 00:28:59.606320 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:59.606192 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13bfa8d3-0ac5-4912-abd6-ef22240f3bc4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13bfa8d3-0ac5-4912-abd6-ef22240f3bc4-plugins-config-volume\") pod \"13bfa8d3-0ac5-4912-abd6-ef22240f3bc4\" (UID: \"13bfa8d3-0ac5-4912-abd6-ef22240f3bc4\") " Nov 03 00:28:59.606635 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:59.606607 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13bfa8d3-0ac5-4912-abd6-ef22240f3bc4-plugins-config-volume" pod "13bfa8d3-0ac5-4912-abd6-ef22240f3bc4" (UID: "13bfa8d3-0ac5-4912-abd6-ef22240f3bc4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:28:59.608513 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:59.608490 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13bfa8d3-0ac5-4912-abd6-ef22240f3bc4-kube-api-access-kj8dp" pod "13bfa8d3-0ac5-4912-abd6-ef22240f3bc4" (UID: "13bfa8d3-0ac5-4912-abd6-ef22240f3bc4"). InnerVolumeSpecName "kube-api-access-kj8dp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:28:59.707536 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:59.707462 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj8dp\" (UniqueName: \"kubernetes.io/projected/13bfa8d3-0ac5-4912-abd6-ef22240f3bc4-kube-api-access-kj8dp\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:28:59.707536 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:59.707489 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13bfa8d3-0ac5-4912-abd6-ef22240f3bc4-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:28:59.882736 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:28:59.882701 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="039c7313-ed6a-400b-85bb-ca3d4352c445" path="/var/lib/kubelet/pods/039c7313-ed6a-400b-85bb-ca3d4352c445/volumes" Nov 03 00:29:00.143637 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:00.143604 5727 generic.go:356] "Generic (PLEG): container finished" podID="13bfa8d3-0ac5-4912-abd6-ef22240f3bc4" containerID="b16e485c44b561ce9f5aa4335d6e41aaea44d171581c56a13a81a445ad5b1e55" exitCode=0 Nov 03 00:29:00.144047 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:00.143648 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-f9cfcc96b-mtll9" Nov 03 00:29:00.144047 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:00.143681 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-f9cfcc96b-mtll9" event={"ID":"13bfa8d3-0ac5-4912-abd6-ef22240f3bc4","Type":"ContainerDied","Data":"b16e485c44b561ce9f5aa4335d6e41aaea44d171581c56a13a81a445ad5b1e55"} Nov 03 00:29:00.144047 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:00.143736 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-f9cfcc96b-mtll9" event={"ID":"13bfa8d3-0ac5-4912-abd6-ef22240f3bc4","Type":"ContainerDied","Data":"16d618cbf51a8bd89607cfd84157e98643e62167cdff1186be55ca73af38c2ba"} Nov 03 00:29:00.144047 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:00.143767 5727 scope.go:122] "RemoveContainer" containerID="b16e485c44b561ce9f5aa4335d6e41aaea44d171581c56a13a81a445ad5b1e55" Nov 03 00:29:00.153860 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:00.153836 5727 scope.go:122] "RemoveContainer" containerID="b16e485c44b561ce9f5aa4335d6e41aaea44d171581c56a13a81a445ad5b1e55" Nov 03 00:29:00.154193 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:29:00.154169 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b16e485c44b561ce9f5aa4335d6e41aaea44d171581c56a13a81a445ad5b1e55\": container with ID starting with b16e485c44b561ce9f5aa4335d6e41aaea44d171581c56a13a81a445ad5b1e55 not found: ID does not exist" containerID="b16e485c44b561ce9f5aa4335d6e41aaea44d171581c56a13a81a445ad5b1e55" Nov 03 00:29:00.154280 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:00.154206 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b16e485c44b561ce9f5aa4335d6e41aaea44d171581c56a13a81a445ad5b1e55"} err="failed to get container status \"b16e485c44b561ce9f5aa4335d6e41aaea44d171581c56a13a81a445ad5b1e55\": rpc error: code = NotFound desc = could not find container \"b16e485c44b561ce9f5aa4335d6e41aaea44d171581c56a13a81a445ad5b1e55\": container with ID starting with b16e485c44b561ce9f5aa4335d6e41aaea44d171581c56a13a81a445ad5b1e55 not found: ID does not exist" Nov 03 00:29:00.308223 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:00.308192 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-f9cfcc96b-mtll9"] Nov 03 00:29:00.313346 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:00.313316 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-f9cfcc96b-mtll9"] Nov 03 00:29:00.981271 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:00.981234 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-95ntp"] Nov 03 00:29:00.981694 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:00.981682 5727 cpu_manager.go:414] "containerMap: removing container" podUID="f36665e4-38c7-4d93-b047-306185b1a17f" containerName="payload-processing" Nov 03 00:29:00.981745 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:00.981696 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f36665e4-38c7-4d93-b047-306185b1a17f" containerName="payload-processing" Nov 03 00:29:00.981745 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:00.981724 5727 cpu_manager.go:414] "containerMap: removing container" podUID="13bfa8d3-0ac5-4912-abd6-ef22240f3bc4" containerName="payload-pre-processing" Nov 03 00:29:00.981745 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:00.981729 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13bfa8d3-0ac5-4912-abd6-ef22240f3bc4" containerName="payload-pre-processing" Nov 03 00:29:00.981745 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:00.981742 5727 cpu_manager.go:414] "containerMap: removing container" podUID="039c7313-ed6a-400b-85bb-ca3d4352c445" containerName="payload-processing" Nov 03 00:29:00.981745 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:00.981746 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="039c7313-ed6a-400b-85bb-ca3d4352c445" containerName="payload-processing" Nov 03 00:29:00.981904 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:00.981831 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="f36665e4-38c7-4d93-b047-306185b1a17f" containerName="payload-processing" Nov 03 00:29:00.981904 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:00.981842 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="13bfa8d3-0ac5-4912-abd6-ef22240f3bc4" containerName="payload-pre-processing" Nov 03 00:29:00.981904 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:00.981850 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="039c7313-ed6a-400b-85bb-ca3d4352c445" containerName="payload-processing" Nov 03 00:29:00.986558 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:00.986540 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-95ntp" Nov 03 00:29:01.089554 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:01.089488 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 00:29:01.089554 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:01.089509 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 00:29:01.089554 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:01.089512 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-m9c62\"" Nov 03 00:29:01.089862 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:01.089509 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\"" Nov 03 00:29:01.089862 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:01.089568 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 00:29:01.089862 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:01.089534 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 00:29:01.092526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:01.092504 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-95ntp"] Nov 03 00:29:01.121969 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:01.121943 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/060291d9-5d0a-4787-982d-235251473cfb-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-95ntp\" (UID: \"060291d9-5d0a-4787-982d-235251473cfb\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-95ntp" Nov 03 00:29:01.122085 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:01.121984 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx8v4\" (UniqueName: \"kubernetes.io/projected/060291d9-5d0a-4787-982d-235251473cfb-kube-api-access-wx8v4\") pod \"payload-processing-external-model-5bcd7997-95ntp\" (UID: \"060291d9-5d0a-4787-982d-235251473cfb\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-95ntp" Nov 03 00:29:01.122154 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:01.122074 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/060291d9-5d0a-4787-982d-235251473cfb-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-95ntp\" (UID: \"060291d9-5d0a-4787-982d-235251473cfb\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-95ntp" Nov 03 00:29:01.122154 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:01.122143 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/060291d9-5d0a-4787-982d-235251473cfb-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-95ntp\" (UID: \"060291d9-5d0a-4787-982d-235251473cfb\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-95ntp" Nov 03 00:29:01.223164 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:01.223109 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx8v4\" (UniqueName: \"kubernetes.io/projected/060291d9-5d0a-4787-982d-235251473cfb-kube-api-access-wx8v4\") pod \"payload-processing-external-model-5bcd7997-95ntp\" (UID: \"060291d9-5d0a-4787-982d-235251473cfb\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-95ntp" Nov 03 00:29:01.223652 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:01.223188 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/060291d9-5d0a-4787-982d-235251473cfb-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-95ntp\" (UID: \"060291d9-5d0a-4787-982d-235251473cfb\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-95ntp" Nov 03 00:29:01.223652 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:01.223219 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/060291d9-5d0a-4787-982d-235251473cfb-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-95ntp\" (UID: \"060291d9-5d0a-4787-982d-235251473cfb\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-95ntp" Nov 03 00:29:01.223652 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:01.223325 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/060291d9-5d0a-4787-982d-235251473cfb-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-95ntp\" (UID: \"060291d9-5d0a-4787-982d-235251473cfb\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-95ntp" Nov 03 00:29:01.224081 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:01.224058 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/060291d9-5d0a-4787-982d-235251473cfb-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-95ntp\" (UID: \"060291d9-5d0a-4787-982d-235251473cfb\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-95ntp" Nov 03 00:29:01.224252 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:01.224229 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/060291d9-5d0a-4787-982d-235251473cfb-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-95ntp\" (UID: \"060291d9-5d0a-4787-982d-235251473cfb\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-95ntp" Nov 03 00:29:01.226352 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:01.226330 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/060291d9-5d0a-4787-982d-235251473cfb-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-95ntp\" (UID: \"060291d9-5d0a-4787-982d-235251473cfb\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-95ntp" Nov 03 00:29:01.283897 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:01.283839 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx8v4\" (UniqueName: \"kubernetes.io/projected/060291d9-5d0a-4787-982d-235251473cfb-kube-api-access-wx8v4\") pod \"payload-processing-external-model-5bcd7997-95ntp\" (UID: \"060291d9-5d0a-4787-982d-235251473cfb\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-95ntp" Nov 03 00:29:01.303561 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:01.303533 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-95ntp" Nov 03 00:29:01.689242 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:01.689210 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-95ntp"] Nov 03 00:29:01.690408 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:29:01.690376 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060291d9_5d0a_4787_982d_235251473cfb.slice/crio-442ef91c91ff7cde504f0d8d6342216b0631fbc4a324c488233521ba0be1f982 WatchSource:0}: Error finding container 442ef91c91ff7cde504f0d8d6342216b0631fbc4a324c488233521ba0be1f982: Status 404 returned error can't find the container with id 442ef91c91ff7cde504f0d8d6342216b0631fbc4a324c488233521ba0be1f982 Nov 03 00:29:01.882519 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:01.882486 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13bfa8d3-0ac5-4912-abd6-ef22240f3bc4" path="/var/lib/kubelet/pods/13bfa8d3-0ac5-4912-abd6-ef22240f3bc4/volumes" Nov 03 00:29:02.162821 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:02.160759 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-95ntp" event={"ID":"060291d9-5d0a-4787-982d-235251473cfb","Type":"ContainerStarted","Data":"e7cfc1cbe4f36ed7d8ede03de2e8476ad76ce08d15e905c4339e01280063d0be"} Nov 03 00:29:02.162821 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:02.160808 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-95ntp" event={"ID":"060291d9-5d0a-4787-982d-235251473cfb","Type":"ContainerStarted","Data":"442ef91c91ff7cde504f0d8d6342216b0631fbc4a324c488233521ba0be1f982"} Nov 03 00:29:02.162821 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:02.160909 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-95ntp" Nov 03 00:29:02.283155 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:02.282985 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-95ntp" podStartSLOduration=2.28296189 podStartE2EDuration="2.28296189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:02.276712199 +0000 UTC m=+6028.515209862" watchObservedRunningTime="2026-11-03 00:29:02.28296189 +0000 UTC m=+6028.521459554" Nov 03 00:29:13.189207 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:13.189175 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-95ntp" Nov 03 00:29:27.844787 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:29:27.844746 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:29:42.048137 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:42.048090 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b823784f-6jtzt"] Nov 03 00:29:42.097164 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:42.097120 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b823784f-6jtzt"] Nov 03 00:29:43.882108 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:29:43.882077 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b14adfd8-b845-4a2d-bf35-f8c45da4343f" path="/var/lib/kubelet/pods/b14adfd8-b845-4a2d-bf35-f8c45da4343f/volumes" Nov 03 00:29:57.845631 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:29:57.845597 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:30:00.170381 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:00.170346 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894430-hw8zn"] Nov 03 00:30:00.174346 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:00.174322 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-hw8zn" Nov 03 00:30:00.189609 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:00.189585 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894430-hw8zn\" (UID: \"198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-hw8zn" Nov 03 00:30:00.189730 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:00.189691 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wwwg\" (UniqueName: \"kubernetes.io/projected/198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc-kube-api-access-4wwwg\") pod \"maas-api-key-cleanup-29894430-hw8zn\" (UID: \"198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-hw8zn" Nov 03 00:30:00.230787 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:00.230759 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894430-hw8zn"] Nov 03 00:30:00.277083 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:00.277055 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:30:00.277083 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:00.277056 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:30:00.277341 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:00.277197 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:30:00.290277 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:00.290250 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wwwg\" (UniqueName: \"kubernetes.io/projected/198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc-kube-api-access-4wwwg\") pod \"maas-api-key-cleanup-29894430-hw8zn\" (UID: \"198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-hw8zn" Nov 03 00:30:00.290429 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:00.290334 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894430-hw8zn\" (UID: \"198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-hw8zn" Nov 03 00:30:00.290978 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:00.290957 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894430-hw8zn\" (UID: \"198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-hw8zn" Nov 03 00:30:00.352939 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:00.352907 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wwwg\" (UniqueName: \"kubernetes.io/projected/198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc-kube-api-access-4wwwg\") pod \"maas-api-key-cleanup-29894430-hw8zn\" (UID: \"198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-hw8zn" Nov 03 00:30:00.487396 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:00.487316 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-hw8zn" Nov 03 00:30:00.862923 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:00.862897 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894430-hw8zn"] Nov 03 00:30:00.864593 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:30:00.864557 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod198c4b9b_6a6b_46bf_a328_d9ec9a84c9cc.slice/crio-f150a2a2992f113d3917de3e89b47babebda7f3c435d4656324b49b659957f17 WatchSource:0}: Error finding container f150a2a2992f113d3917de3e89b47babebda7f3c435d4656324b49b659957f17: Status 404 returned error can't find the container with id f150a2a2992f113d3917de3e89b47babebda7f3c435d4656324b49b659957f17 Nov 03 00:30:01.555337 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:01.555250 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-hw8zn" event={"ID":"198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc","Type":"ContainerStarted","Data":"4f317ff77ca7dc7eaeb6d1b52a1df377da30ed83cf458d0476b0f53b03c2ae7c"} Nov 03 00:30:01.555337 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:01.555307 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-hw8zn" event={"ID":"198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc","Type":"ContainerStarted","Data":"f150a2a2992f113d3917de3e89b47babebda7f3c435d4656324b49b659957f17"} Nov 03 00:30:01.674503 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:01.674443 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-hw8zn" podStartSLOduration=1.674424341 podStartE2EDuration="1.674424341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:01.670215525 +0000 UTC m=+6087.908713187" watchObservedRunningTime="2026-11-03 00:30:01.674424341 +0000 UTC m=+6087.912922006" Nov 03 00:30:02.563087 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:02.563045 5727 generic.go:356] "Generic (PLEG): container finished" podID="198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc" containerID="4f317ff77ca7dc7eaeb6d1b52a1df377da30ed83cf458d0476b0f53b03c2ae7c" exitCode=60 Nov 03 00:30:02.563548 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:02.563109 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-hw8zn" event={"ID":"198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc","Type":"ContainerDied","Data":"4f317ff77ca7dc7eaeb6d1b52a1df377da30ed83cf458d0476b0f53b03c2ae7c"} Nov 03 00:30:02.563548 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:02.563402 5727 scope.go:122] "RemoveContainer" containerID="4f317ff77ca7dc7eaeb6d1b52a1df377da30ed83cf458d0476b0f53b03c2ae7c" Nov 03 00:30:03.570560 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:03.570526 5727 generic.go:356] "Generic (PLEG): container finished" podID="198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc" containerID="2baab7cf8f545f4631f1015f83dd13b0b2d71da237f185e585d187fe259f1b4c" exitCode=60 Nov 03 00:30:03.571061 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:03.570597 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-hw8zn" event={"ID":"198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc","Type":"ContainerDied","Data":"2baab7cf8f545f4631f1015f83dd13b0b2d71da237f185e585d187fe259f1b4c"} Nov 03 00:30:03.571061 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:03.570668 5727 scope.go:122] "RemoveContainer" containerID="4f317ff77ca7dc7eaeb6d1b52a1df377da30ed83cf458d0476b0f53b03c2ae7c" Nov 03 00:30:03.571061 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:03.570875 5727 scope.go:122] "RemoveContainer" containerID="2baab7cf8f545f4631f1015f83dd13b0b2d71da237f185e585d187fe259f1b4c" Nov 03 00:30:03.571225 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:30:03.571158 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894430-hw8zn_odh-ai-gateway-infra(198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-hw8zn" podUID="198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc" Nov 03 00:30:04.579140 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:04.579103 5727 scope.go:122] "RemoveContainer" containerID="2baab7cf8f545f4631f1015f83dd13b0b2d71da237f185e585d187fe259f1b4c" Nov 03 00:30:04.579550 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:30:04.579374 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894430-hw8zn_odh-ai-gateway-infra(198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-hw8zn" podUID="198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc" Nov 03 00:30:15.875909 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:15.875876 5727 scope.go:122] "RemoveContainer" containerID="2baab7cf8f545f4631f1015f83dd13b0b2d71da237f185e585d187fe259f1b4c" Nov 03 00:30:16.664796 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:16.664760 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-hw8zn" event={"ID":"198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc","Type":"ContainerStarted","Data":"bf180655aa13a6233175a79d2bc5867d0ecd6e2a7ede0ec51c0ebe2bea3db10b"} Nov 03 00:30:17.674460 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:17.674421 5727 generic.go:356] "Generic (PLEG): container finished" podID="198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc" containerID="bf180655aa13a6233175a79d2bc5867d0ecd6e2a7ede0ec51c0ebe2bea3db10b" exitCode=60 Nov 03 00:30:17.674958 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:17.674483 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-hw8zn" event={"ID":"198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc","Type":"ContainerDied","Data":"bf180655aa13a6233175a79d2bc5867d0ecd6e2a7ede0ec51c0ebe2bea3db10b"} Nov 03 00:30:17.674958 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:17.674530 5727 scope.go:122] "RemoveContainer" containerID="2baab7cf8f545f4631f1015f83dd13b0b2d71da237f185e585d187fe259f1b4c" Nov 03 00:30:17.674958 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:17.674913 5727 scope.go:122] "RemoveContainer" containerID="bf180655aa13a6233175a79d2bc5867d0ecd6e2a7ede0ec51c0ebe2bea3db10b" Nov 03 00:30:17.675297 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:30:17.675273 5727 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894430-hw8zn_odh-ai-gateway-infra(198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-hw8zn" podUID="198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc" Nov 03 00:30:17.790747 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:17.790710 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894430-hw8zn"] Nov 03 00:30:18.824841 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:18.824817 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-hw8zn" Nov 03 00:30:18.876561 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:18.876534 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc-kube-api-access-4wwwg\" (UniqueName: \"kubernetes.io/projected/198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc-kube-api-access-4wwwg\") pod \"198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc\" (UID: \"198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc\") " Nov 03 00:30:18.876681 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:18.876583 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc-maas-api-service-ca\") pod \"198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc\" (UID: \"198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc\") " Nov 03 00:30:18.876926 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:18.876905 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc-maas-api-service-ca" pod "198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc" (UID: "198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:30:18.878808 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:18.878787 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc-kube-api-access-4wwwg" pod "198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc" (UID: "198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc"). InnerVolumeSpecName "kube-api-access-4wwwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:18.978246 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:18.978164 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wwwg\" (UniqueName: \"kubernetes.io/projected/198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc-kube-api-access-4wwwg\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:18.978246 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:18.978198 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc-maas-api-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:19.688913 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:19.688881 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-hw8zn" Nov 03 00:30:19.688913 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:19.688909 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-hw8zn" event={"ID":"198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc","Type":"ContainerDied","Data":"f150a2a2992f113d3917de3e89b47babebda7f3c435d4656324b49b659957f17"} Nov 03 00:30:19.689211 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:19.688968 5727 scope.go:122] "RemoveContainer" containerID="bf180655aa13a6233175a79d2bc5867d0ecd6e2a7ede0ec51c0ebe2bea3db10b" Nov 03 00:30:19.852482 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:19.852449 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894430-hw8zn"] Nov 03 00:30:19.901939 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:19.901910 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894430-hw8zn"] Nov 03 00:30:21.881951 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:21.881915 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc" path="/var/lib/kubelet/pods/198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc/volumes" Nov 03 00:30:27.846282 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:30:27.846241 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:30:45.140534 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:45.140448 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-29fsr"] Nov 03 00:30:45.140984 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:45.140911 5727 cpu_manager.go:414] "containerMap: removing container" podUID="198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc" containerName="cleanup" Nov 03 00:30:45.140984 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:45.140922 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc" containerName="cleanup" Nov 03 00:30:45.140984 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:45.140943 5727 cpu_manager.go:414] "containerMap: removing container" podUID="198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc" containerName="cleanup" Nov 03 00:30:45.140984 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:45.140948 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc" containerName="cleanup" Nov 03 00:30:45.141181 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:45.141024 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc" containerName="cleanup" Nov 03 00:30:45.141181 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:45.141039 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc" containerName="cleanup" Nov 03 00:30:45.145777 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:45.145759 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-29fsr" Nov 03 00:30:45.198321 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:45.198271 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:30:45.202122 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:45.202086 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-29fsr"] Nov 03 00:30:45.231074 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:45.231039 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zj9g\" (UniqueName: \"kubernetes.io/projected/c8a714c1-5619-4456-863a-cf9c0e1a250a-kube-api-access-7zj9g\") pod \"maas-api-revoke-sub-e2547622-29fsr\" (UID: \"c8a714c1-5619-4456-863a-cf9c0e1a250a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-29fsr" Nov 03 00:30:45.231245 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:45.231110 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8a714c1-5619-4456-863a-cf9c0e1a250a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-29fsr\" (UID: \"c8a714c1-5619-4456-863a-cf9c0e1a250a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-29fsr" Nov 03 00:30:45.231352 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:45.231331 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8a714c1-5619-4456-863a-cf9c0e1a250a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-29fsr\" (UID: \"c8a714c1-5619-4456-863a-cf9c0e1a250a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-29fsr" Nov 03 00:30:45.247663 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:45.247631 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:30:45.247857 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:45.247841 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:30:45.247928 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:45.247865 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:30:45.331935 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:45.331903 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zj9g\" (UniqueName: \"kubernetes.io/projected/c8a714c1-5619-4456-863a-cf9c0e1a250a-kube-api-access-7zj9g\") pod \"maas-api-revoke-sub-e2547622-29fsr\" (UID: \"c8a714c1-5619-4456-863a-cf9c0e1a250a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-29fsr" Nov 03 00:30:45.332116 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:45.331952 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8a714c1-5619-4456-863a-cf9c0e1a250a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-29fsr\" (UID: \"c8a714c1-5619-4456-863a-cf9c0e1a250a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-29fsr" Nov 03 00:30:45.332116 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:45.332031 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8a714c1-5619-4456-863a-cf9c0e1a250a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-29fsr\" (UID: \"c8a714c1-5619-4456-863a-cf9c0e1a250a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-29fsr" Nov 03 00:30:45.332663 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:45.332644 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8a714c1-5619-4456-863a-cf9c0e1a250a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-29fsr\" (UID: \"c8a714c1-5619-4456-863a-cf9c0e1a250a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-29fsr" Nov 03 00:30:45.334494 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:45.334470 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8a714c1-5619-4456-863a-cf9c0e1a250a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-29fsr\" (UID: \"c8a714c1-5619-4456-863a-cf9c0e1a250a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-29fsr" Nov 03 00:30:45.392733 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:45.392664 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zj9g\" (UniqueName: \"kubernetes.io/projected/c8a714c1-5619-4456-863a-cf9c0e1a250a-kube-api-access-7zj9g\") pod \"maas-api-revoke-sub-e2547622-29fsr\" (UID: \"c8a714c1-5619-4456-863a-cf9c0e1a250a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-29fsr" Nov 03 00:30:45.459939 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:45.459907 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-29fsr" Nov 03 00:30:45.834963 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:45.834935 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-29fsr"] Nov 03 00:30:45.836910 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:30:45.836881 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a714c1_5619_4456_863a_cf9c0e1a250a.slice/crio-cdd03273d918b0ecbaa5f7f285860f956ba9a23db7b68bedb958865e11fcf501 WatchSource:0}: Error finding container cdd03273d918b0ecbaa5f7f285860f956ba9a23db7b68bedb958865e11fcf501: Status 404 returned error can't find the container with id cdd03273d918b0ecbaa5f7f285860f956ba9a23db7b68bedb958865e11fcf501 Nov 03 00:30:45.859804 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:45.859771 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-29fsr" event={"ID":"c8a714c1-5619-4456-863a-cf9c0e1a250a","Type":"ContainerStarted","Data":"cdd03273d918b0ecbaa5f7f285860f956ba9a23db7b68bedb958865e11fcf501"} Nov 03 00:30:46.720181 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:30:46.720146 5727 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a714c1_5619_4456_863a_cf9c0e1a250a.slice/crio-8f604c17966c27d8c4e8c75cfbc384a6e7904b44de9274b71c2a09fa118a27a7.scope\": RecentStats: unable to find data in memory cache]" Nov 03 00:30:46.720522 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:30:46.720295 5727 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a714c1_5619_4456_863a_cf9c0e1a250a.slice/crio-8f604c17966c27d8c4e8c75cfbc384a6e7904b44de9274b71c2a09fa118a27a7.scope\": RecentStats: unable to find data in memory cache]" Nov 03 00:30:46.868496 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:46.868453 5727 generic.go:356] "Generic (PLEG): container finished" podID="c8a714c1-5619-4456-863a-cf9c0e1a250a" containerID="8f604c17966c27d8c4e8c75cfbc384a6e7904b44de9274b71c2a09fa118a27a7" exitCode=0 Nov 03 00:30:46.868716 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:46.868525 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-29fsr" event={"ID":"c8a714c1-5619-4456-863a-cf9c0e1a250a","Type":"ContainerDied","Data":"8f604c17966c27d8c4e8c75cfbc384a6e7904b44de9274b71c2a09fa118a27a7"} Nov 03 00:30:48.030023 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:48.029996 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-29fsr" Nov 03 00:30:48.054203 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:48.054176 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8a714c1-5619-4456-863a-cf9c0e1a250a-kube-api-access-7zj9g\" (UniqueName: \"kubernetes.io/projected/c8a714c1-5619-4456-863a-cf9c0e1a250a-kube-api-access-7zj9g\") pod \"c8a714c1-5619-4456-863a-cf9c0e1a250a\" (UID: \"c8a714c1-5619-4456-863a-cf9c0e1a250a\") " Nov 03 00:30:48.054365 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:48.054294 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8a714c1-5619-4456-863a-cf9c0e1a250a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8a714c1-5619-4456-863a-cf9c0e1a250a-maas-api-xks-ca\") pod \"c8a714c1-5619-4456-863a-cf9c0e1a250a\" (UID: \"c8a714c1-5619-4456-863a-cf9c0e1a250a\") " Nov 03 00:30:48.054365 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:48.054340 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8a714c1-5619-4456-863a-cf9c0e1a250a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8a714c1-5619-4456-863a-cf9c0e1a250a-maas-api-openshift-ca\") pod \"c8a714c1-5619-4456-863a-cf9c0e1a250a\" (UID: \"c8a714c1-5619-4456-863a-cf9c0e1a250a\") " Nov 03 00:30:48.054698 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:48.054672 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8a714c1-5619-4456-863a-cf9c0e1a250a-maas-api-openshift-ca" pod "c8a714c1-5619-4456-863a-cf9c0e1a250a" (UID: "c8a714c1-5619-4456-863a-cf9c0e1a250a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:30:48.056529 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:48.056506 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a714c1-5619-4456-863a-cf9c0e1a250a-kube-api-access-7zj9g" pod "c8a714c1-5619-4456-863a-cf9c0e1a250a" (UID: "c8a714c1-5619-4456-863a-cf9c0e1a250a"). InnerVolumeSpecName "kube-api-access-7zj9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:48.056620 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:48.056568 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8a714c1-5619-4456-863a-cf9c0e1a250a-maas-api-xks-ca" pod "c8a714c1-5619-4456-863a-cf9c0e1a250a" (UID: "c8a714c1-5619-4456-863a-cf9c0e1a250a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:30:48.155518 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:48.155442 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8a714c1-5619-4456-863a-cf9c0e1a250a-maas-api-xks-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:48.155518 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:48.155470 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8a714c1-5619-4456-863a-cf9c0e1a250a-maas-api-openshift-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:48.155518 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:48.155480 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zj9g\" (UniqueName: \"kubernetes.io/projected/c8a714c1-5619-4456-863a-cf9c0e1a250a-kube-api-access-7zj9g\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:48.882030 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:48.881996 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-29fsr" event={"ID":"c8a714c1-5619-4456-863a-cf9c0e1a250a","Type":"ContainerDied","Data":"cdd03273d918b0ecbaa5f7f285860f956ba9a23db7b68bedb958865e11fcf501"} Nov 03 00:30:48.882233 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:48.882035 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-29fsr" Nov 03 00:30:48.882233 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:48.882040 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdd03273d918b0ecbaa5f7f285860f956ba9a23db7b68bedb958865e11fcf501" Nov 03 00:30:56.669852 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:30:56.669819 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-95ntp"] Nov 03 00:30:57.851663 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:30:57.851621 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:31:01.673267 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:01.673193 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-95ntp" podUID="060291d9-5d0a-4787-982d-235251473cfb" containerName="payload-processing" containerID="cri-o://e7cfc1cbe4f36ed7d8ede03de2e8476ad76ce08d15e905c4339e01280063d0be" gracePeriod=25 Nov 03 00:31:01.995647 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:01.995562 5727 generic.go:356] "Generic (PLEG): container finished" podID="060291d9-5d0a-4787-982d-235251473cfb" containerID="e7cfc1cbe4f36ed7d8ede03de2e8476ad76ce08d15e905c4339e01280063d0be" exitCode=0 Nov 03 00:31:01.996028 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:01.995940 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-95ntp" event={"ID":"060291d9-5d0a-4787-982d-235251473cfb","Type":"ContainerDied","Data":"e7cfc1cbe4f36ed7d8ede03de2e8476ad76ce08d15e905c4339e01280063d0be"} Nov 03 00:31:02.089711 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:02.089684 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-95ntp" Nov 03 00:31:02.113715 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:02.113689 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/060291d9-5d0a-4787-982d-235251473cfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/060291d9-5d0a-4787-982d-235251473cfb-plugins-config-volume\") pod \"060291d9-5d0a-4787-982d-235251473cfb\" (UID: \"060291d9-5d0a-4787-982d-235251473cfb\") " Nov 03 00:31:02.113859 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:02.113779 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/060291d9-5d0a-4787-982d-235251473cfb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/060291d9-5d0a-4787-982d-235251473cfb-routing-overlay-volume\") pod \"060291d9-5d0a-4787-982d-235251473cfb\" (UID: \"060291d9-5d0a-4787-982d-235251473cfb\") " Nov 03 00:31:02.113859 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:02.113823 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/060291d9-5d0a-4787-982d-235251473cfb-kube-api-access-wx8v4\" (UniqueName: \"kubernetes.io/projected/060291d9-5d0a-4787-982d-235251473cfb-kube-api-access-wx8v4\") pod \"060291d9-5d0a-4787-982d-235251473cfb\" (UID: \"060291d9-5d0a-4787-982d-235251473cfb\") " Nov 03 00:31:02.113859 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:02.113845 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/060291d9-5d0a-4787-982d-235251473cfb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/060291d9-5d0a-4787-982d-235251473cfb-provider-credentials\") pod \"060291d9-5d0a-4787-982d-235251473cfb\" (UID: \"060291d9-5d0a-4787-982d-235251473cfb\") " Nov 03 00:31:02.114158 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:02.114118 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/060291d9-5d0a-4787-982d-235251473cfb-plugins-config-volume" pod "060291d9-5d0a-4787-982d-235251473cfb" (UID: "060291d9-5d0a-4787-982d-235251473cfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:31:02.114265 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:02.114212 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/060291d9-5d0a-4787-982d-235251473cfb-routing-overlay-volume" pod "060291d9-5d0a-4787-982d-235251473cfb" (UID: "060291d9-5d0a-4787-982d-235251473cfb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:31:02.116264 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:02.116238 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060291d9-5d0a-4787-982d-235251473cfb-provider-credentials" pod "060291d9-5d0a-4787-982d-235251473cfb" (UID: "060291d9-5d0a-4787-982d-235251473cfb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:02.116362 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:02.116266 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060291d9-5d0a-4787-982d-235251473cfb-kube-api-access-wx8v4" pod "060291d9-5d0a-4787-982d-235251473cfb" (UID: "060291d9-5d0a-4787-982d-235251473cfb"). InnerVolumeSpecName "kube-api-access-wx8v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:02.215365 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:02.215321 5727 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/060291d9-5d0a-4787-982d-235251473cfb-routing-overlay-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:02.215365 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:02.215362 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx8v4\" (UniqueName: \"kubernetes.io/projected/060291d9-5d0a-4787-982d-235251473cfb-kube-api-access-wx8v4\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:02.215541 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:02.215375 5727 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/060291d9-5d0a-4787-982d-235251473cfb-provider-credentials\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:02.215541 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:02.215388 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/060291d9-5d0a-4787-982d-235251473cfb-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:03.004518 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:03.004491 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-95ntp" Nov 03 00:31:03.004975 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:03.004532 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-95ntp" event={"ID":"060291d9-5d0a-4787-982d-235251473cfb","Type":"ContainerDied","Data":"442ef91c91ff7cde504f0d8d6342216b0631fbc4a324c488233521ba0be1f982"} Nov 03 00:31:03.004975 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:03.004594 5727 scope.go:122] "RemoveContainer" containerID="e7cfc1cbe4f36ed7d8ede03de2e8476ad76ce08d15e905c4339e01280063d0be" Nov 03 00:31:03.167888 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:03.167851 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-95ntp"] Nov 03 00:31:03.217297 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:03.217258 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-95ntp"] Nov 03 00:31:03.883795 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:03.883758 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="060291d9-5d0a-4787-982d-235251473cfb" path="/var/lib/kubelet/pods/060291d9-5d0a-4787-982d-235251473cfb/volumes" Nov 03 00:31:12.859261 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:12.859218 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-gcxrh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:31:12.860025 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:12.859991 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" podUID="c4c2f4f5-743d-4e9d-ae63-1769c521b4ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:31:16.145849 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:16.145814 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qjhsz"] Nov 03 00:31:16.146298 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:16.146288 5727 cpu_manager.go:414] "containerMap: removing container" podUID="198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc" containerName="cleanup" Nov 03 00:31:16.146342 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:16.146300 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc" containerName="cleanup" Nov 03 00:31:16.146342 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:16.146316 5727 cpu_manager.go:414] "containerMap: removing container" podUID="c8a714c1-5619-4456-863a-cf9c0e1a250a" containerName="revoke-keys" Nov 03 00:31:16.146342 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:16.146321 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a714c1-5619-4456-863a-cf9c0e1a250a" containerName="revoke-keys" Nov 03 00:31:16.146342 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:16.146331 5727 cpu_manager.go:414] "containerMap: removing container" podUID="060291d9-5d0a-4787-982d-235251473cfb" containerName="payload-processing" Nov 03 00:31:16.146342 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:16.146338 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="060291d9-5d0a-4787-982d-235251473cfb" containerName="payload-processing" Nov 03 00:31:16.146506 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:16.146448 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="198c4b9b-6a6b-46bf-a328-d9ec9a84c9cc" containerName="cleanup" Nov 03 00:31:16.146506 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:16.146458 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8a714c1-5619-4456-863a-cf9c0e1a250a" containerName="revoke-keys" Nov 03 00:31:16.146506 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:16.146465 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="060291d9-5d0a-4787-982d-235251473cfb" containerName="payload-processing" Nov 03 00:31:16.151167 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:16.151146 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qjhsz" Nov 03 00:31:16.176524 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:16.176501 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/468a0873-51e8-4b7f-bf54-5131fd0352da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-qjhsz\" (UID: \"468a0873-51e8-4b7f-bf54-5131fd0352da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qjhsz" Nov 03 00:31:16.176712 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:16.176557 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88s8n\" (UniqueName: \"kubernetes.io/projected/468a0873-51e8-4b7f-bf54-5131fd0352da-kube-api-access-88s8n\") pod \"maas-api-revoke-sub-c1e271a9-qjhsz\" (UID: \"468a0873-51e8-4b7f-bf54-5131fd0352da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qjhsz" Nov 03 00:31:16.176712 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:16.176671 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/468a0873-51e8-4b7f-bf54-5131fd0352da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-qjhsz\" (UID: \"468a0873-51e8-4b7f-bf54-5131fd0352da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qjhsz" Nov 03 00:31:16.206593 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:16.206563 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qjhsz"] Nov 03 00:31:16.252535 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:16.252499 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:31:16.252535 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:16.252505 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:31:16.252535 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:16.252505 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:31:16.252764 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:16.252589 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:31:16.278228 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:16.278202 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88s8n\" (UniqueName: \"kubernetes.io/projected/468a0873-51e8-4b7f-bf54-5131fd0352da-kube-api-access-88s8n\") pod \"maas-api-revoke-sub-c1e271a9-qjhsz\" (UID: \"468a0873-51e8-4b7f-bf54-5131fd0352da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qjhsz" Nov 03 00:31:16.278369 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:16.278247 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/468a0873-51e8-4b7f-bf54-5131fd0352da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-qjhsz\" (UID: \"468a0873-51e8-4b7f-bf54-5131fd0352da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qjhsz" Nov 03 00:31:16.278369 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:16.278315 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/468a0873-51e8-4b7f-bf54-5131fd0352da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-qjhsz\" (UID: \"468a0873-51e8-4b7f-bf54-5131fd0352da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qjhsz" Nov 03 00:31:16.278913 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:16.278891 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/468a0873-51e8-4b7f-bf54-5131fd0352da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-qjhsz\" (UID: \"468a0873-51e8-4b7f-bf54-5131fd0352da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qjhsz" Nov 03 00:31:16.280823 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:16.280800 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/468a0873-51e8-4b7f-bf54-5131fd0352da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-qjhsz\" (UID: \"468a0873-51e8-4b7f-bf54-5131fd0352da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qjhsz" Nov 03 00:31:16.339031 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:16.339002 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88s8n\" (UniqueName: \"kubernetes.io/projected/468a0873-51e8-4b7f-bf54-5131fd0352da-kube-api-access-88s8n\") pod \"maas-api-revoke-sub-c1e271a9-qjhsz\" (UID: \"468a0873-51e8-4b7f-bf54-5131fd0352da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qjhsz" Nov 03 00:31:16.467907 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:16.467823 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qjhsz" Nov 03 00:31:16.845313 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:16.845286 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qjhsz"] Nov 03 00:31:16.846926 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:31:16.846893 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468a0873_51e8_4b7f_bf54_5131fd0352da.slice/crio-73341a2b2e6cc25725da299c238f9c0b3bd13842e96095a93665ab4bf5d07806 WatchSource:0}: Error finding container 73341a2b2e6cc25725da299c238f9c0b3bd13842e96095a93665ab4bf5d07806: Status 404 returned error can't find the container with id 73341a2b2e6cc25725da299c238f9c0b3bd13842e96095a93665ab4bf5d07806 Nov 03 00:31:17.153442 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:17.153406 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qjhsz" event={"ID":"468a0873-51e8-4b7f-bf54-5131fd0352da","Type":"ContainerStarted","Data":"73341a2b2e6cc25725da299c238f9c0b3bd13842e96095a93665ab4bf5d07806"} Nov 03 00:31:18.161971 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:18.161930 5727 generic.go:356] "Generic (PLEG): container finished" podID="468a0873-51e8-4b7f-bf54-5131fd0352da" containerID="8572dfc9be53e189cfce50d3ded3f846451e4dc8d1f35e3b14dda3bc3e1d9c1d" exitCode=0 Nov 03 00:31:18.162402 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:18.162009 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qjhsz" event={"ID":"468a0873-51e8-4b7f-bf54-5131fd0352da","Type":"ContainerDied","Data":"8572dfc9be53e189cfce50d3ded3f846451e4dc8d1f35e3b14dda3bc3e1d9c1d"} Nov 03 00:31:19.313623 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:19.313594 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qjhsz" Nov 03 00:31:19.409521 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:19.409486 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/468a0873-51e8-4b7f-bf54-5131fd0352da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/468a0873-51e8-4b7f-bf54-5131fd0352da-maas-api-openshift-ca\") pod \"468a0873-51e8-4b7f-bf54-5131fd0352da\" (UID: \"468a0873-51e8-4b7f-bf54-5131fd0352da\") " Nov 03 00:31:19.409691 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:19.409580 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/468a0873-51e8-4b7f-bf54-5131fd0352da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/468a0873-51e8-4b7f-bf54-5131fd0352da-maas-api-xks-ca\") pod \"468a0873-51e8-4b7f-bf54-5131fd0352da\" (UID: \"468a0873-51e8-4b7f-bf54-5131fd0352da\") " Nov 03 00:31:19.409691 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:19.409672 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/468a0873-51e8-4b7f-bf54-5131fd0352da-kube-api-access-88s8n\" (UniqueName: \"kubernetes.io/projected/468a0873-51e8-4b7f-bf54-5131fd0352da-kube-api-access-88s8n\") pod \"468a0873-51e8-4b7f-bf54-5131fd0352da\" (UID: \"468a0873-51e8-4b7f-bf54-5131fd0352da\") " Nov 03 00:31:19.409848 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:19.409824 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/468a0873-51e8-4b7f-bf54-5131fd0352da-maas-api-openshift-ca" pod "468a0873-51e8-4b7f-bf54-5131fd0352da" (UID: "468a0873-51e8-4b7f-bf54-5131fd0352da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:31:19.410003 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:19.409985 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/468a0873-51e8-4b7f-bf54-5131fd0352da-maas-api-openshift-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:19.411851 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:19.411821 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/468a0873-51e8-4b7f-bf54-5131fd0352da-maas-api-xks-ca" pod "468a0873-51e8-4b7f-bf54-5131fd0352da" (UID: "468a0873-51e8-4b7f-bf54-5131fd0352da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:31:19.411963 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:19.411855 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/468a0873-51e8-4b7f-bf54-5131fd0352da-kube-api-access-88s8n" pod "468a0873-51e8-4b7f-bf54-5131fd0352da" (UID: "468a0873-51e8-4b7f-bf54-5131fd0352da"). InnerVolumeSpecName "kube-api-access-88s8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:19.511050 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:19.510977 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/468a0873-51e8-4b7f-bf54-5131fd0352da-maas-api-xks-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:19.511050 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:19.511003 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88s8n\" (UniqueName: \"kubernetes.io/projected/468a0873-51e8-4b7f-bf54-5131fd0352da-kube-api-access-88s8n\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:20.176052 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:20.175970 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qjhsz" Nov 03 00:31:20.176052 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:20.176012 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qjhsz" event={"ID":"468a0873-51e8-4b7f-bf54-5131fd0352da","Type":"ContainerDied","Data":"73341a2b2e6cc25725da299c238f9c0b3bd13842e96095a93665ab4bf5d07806"} Nov 03 00:31:20.176255 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:20.176057 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73341a2b2e6cc25725da299c238f9c0b3bd13842e96095a93665ab4bf5d07806" Nov 03 00:31:27.845452 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:31:27.845416 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:31:36.248402 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:36.248370 5727 scope.go:122] "RemoveContainer" containerID="31a22d7224ef58ebcc0de602abcfd7e472d1e13156d474291f5827933b5660dc" Nov 03 00:31:41.158168 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:41.158112 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z424d"] Nov 03 00:31:41.158877 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:41.158856 5727 cpu_manager.go:414] "containerMap: removing container" podUID="468a0873-51e8-4b7f-bf54-5131fd0352da" containerName="revoke-keys" Nov 03 00:31:41.158958 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:41.158878 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="468a0873-51e8-4b7f-bf54-5131fd0352da" containerName="revoke-keys" Nov 03 00:31:41.159048 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:41.159036 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="468a0873-51e8-4b7f-bf54-5131fd0352da" containerName="revoke-keys" Nov 03 00:31:41.164331 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:41.164309 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z424d" Nov 03 00:31:41.192622 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:41.192591 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5624412f-4ebe-40ad-835c-0df126ea4729-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-z424d\" (UID: \"5624412f-4ebe-40ad-835c-0df126ea4729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z424d" Nov 03 00:31:41.192777 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:41.192735 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5624412f-4ebe-40ad-835c-0df126ea4729-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-z424d\" (UID: \"5624412f-4ebe-40ad-835c-0df126ea4729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z424d" Nov 03 00:31:41.192852 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:41.192811 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2jwr\" (UniqueName: \"kubernetes.io/projected/5624412f-4ebe-40ad-835c-0df126ea4729-kube-api-access-d2jwr\") pod \"maas-api-revoke-sub-2e1f5b5f-z424d\" (UID: \"5624412f-4ebe-40ad-835c-0df126ea4729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z424d" Nov 03 00:31:41.217989 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:41.217957 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z424d"] Nov 03 00:31:41.263626 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:41.263590 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:31:41.263787 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:41.263634 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:31:41.263787 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:41.263652 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:31:41.263787 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:41.263684 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:31:41.293532 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:41.293507 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5624412f-4ebe-40ad-835c-0df126ea4729-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-z424d\" (UID: \"5624412f-4ebe-40ad-835c-0df126ea4729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z424d" Nov 03 00:31:41.293664 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:41.293574 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5624412f-4ebe-40ad-835c-0df126ea4729-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-z424d\" (UID: \"5624412f-4ebe-40ad-835c-0df126ea4729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z424d" Nov 03 00:31:41.293664 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:41.293604 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2jwr\" (UniqueName: \"kubernetes.io/projected/5624412f-4ebe-40ad-835c-0df126ea4729-kube-api-access-d2jwr\") pod \"maas-api-revoke-sub-2e1f5b5f-z424d\" (UID: \"5624412f-4ebe-40ad-835c-0df126ea4729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z424d" Nov 03 00:31:41.294269 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:41.294244 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5624412f-4ebe-40ad-835c-0df126ea4729-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-z424d\" (UID: \"5624412f-4ebe-40ad-835c-0df126ea4729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z424d" Nov 03 00:31:41.296069 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:41.296049 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5624412f-4ebe-40ad-835c-0df126ea4729-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-z424d\" (UID: \"5624412f-4ebe-40ad-835c-0df126ea4729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z424d" Nov 03 00:31:41.354452 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:41.354416 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2jwr\" (UniqueName: \"kubernetes.io/projected/5624412f-4ebe-40ad-835c-0df126ea4729-kube-api-access-d2jwr\") pod \"maas-api-revoke-sub-2e1f5b5f-z424d\" (UID: \"5624412f-4ebe-40ad-835c-0df126ea4729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z424d" Nov 03 00:31:41.478798 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:41.478721 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z424d" Nov 03 00:31:41.854011 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:41.853979 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z424d"] Nov 03 00:31:41.855260 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:31:41.855233 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5624412f_4ebe_40ad_835c_0df126ea4729.slice/crio-805f3940eced0e7b5e8c8e3bd41612fbd2d5fecf9059ea295bbde30fc9401b60 WatchSource:0}: Error finding container 805f3940eced0e7b5e8c8e3bd41612fbd2d5fecf9059ea295bbde30fc9401b60: Status 404 returned error can't find the container with id 805f3940eced0e7b5e8c8e3bd41612fbd2d5fecf9059ea295bbde30fc9401b60 Nov 03 00:31:42.369433 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:42.369395 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z424d" event={"ID":"5624412f-4ebe-40ad-835c-0df126ea4729","Type":"ContainerStarted","Data":"10cdf98dfeda9d4c38e0981e32f60df469c337103c5e1a1ff5c232b48d91017b"} Nov 03 00:31:42.369833 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:42.369452 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z424d" event={"ID":"5624412f-4ebe-40ad-835c-0df126ea4729","Type":"ContainerStarted","Data":"805f3940eced0e7b5e8c8e3bd41612fbd2d5fecf9059ea295bbde30fc9401b60"} Nov 03 00:31:42.486472 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:42.486371 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z424d" podStartSLOduration=1.486357898 podStartE2EDuration="1.486357898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:31:42.485084985 +0000 UTC m=+6188.723582659" watchObservedRunningTime="2026-11-03 00:31:42.486357898 +0000 UTC m=+6188.724855560" Nov 03 00:31:43.377762 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:43.377722 5727 generic.go:356] "Generic (PLEG): container finished" podID="5624412f-4ebe-40ad-835c-0df126ea4729" containerID="10cdf98dfeda9d4c38e0981e32f60df469c337103c5e1a1ff5c232b48d91017b" exitCode=0 Nov 03 00:31:43.378240 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:43.377818 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z424d" event={"ID":"5624412f-4ebe-40ad-835c-0df126ea4729","Type":"ContainerDied","Data":"10cdf98dfeda9d4c38e0981e32f60df469c337103c5e1a1ff5c232b48d91017b"} Nov 03 00:31:44.533221 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:44.533195 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z424d" Nov 03 00:31:44.625319 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:44.625291 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5624412f-4ebe-40ad-835c-0df126ea4729-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5624412f-4ebe-40ad-835c-0df126ea4729-maas-api-xks-ca\") pod \"5624412f-4ebe-40ad-835c-0df126ea4729\" (UID: \"5624412f-4ebe-40ad-835c-0df126ea4729\") " Nov 03 00:31:44.625454 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:44.625328 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5624412f-4ebe-40ad-835c-0df126ea4729-kube-api-access-d2jwr\" (UniqueName: \"kubernetes.io/projected/5624412f-4ebe-40ad-835c-0df126ea4729-kube-api-access-d2jwr\") pod \"5624412f-4ebe-40ad-835c-0df126ea4729\" (UID: \"5624412f-4ebe-40ad-835c-0df126ea4729\") " Nov 03 00:31:44.625454 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:44.625351 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5624412f-4ebe-40ad-835c-0df126ea4729-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5624412f-4ebe-40ad-835c-0df126ea4729-maas-api-openshift-ca\") pod \"5624412f-4ebe-40ad-835c-0df126ea4729\" (UID: \"5624412f-4ebe-40ad-835c-0df126ea4729\") " Nov 03 00:31:44.625769 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:44.625745 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5624412f-4ebe-40ad-835c-0df126ea4729-maas-api-openshift-ca" pod "5624412f-4ebe-40ad-835c-0df126ea4729" (UID: "5624412f-4ebe-40ad-835c-0df126ea4729"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:31:44.627666 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:44.627642 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5624412f-4ebe-40ad-835c-0df126ea4729-maas-api-xks-ca" pod "5624412f-4ebe-40ad-835c-0df126ea4729" (UID: "5624412f-4ebe-40ad-835c-0df126ea4729"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:31:44.627745 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:44.627663 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5624412f-4ebe-40ad-835c-0df126ea4729-kube-api-access-d2jwr" pod "5624412f-4ebe-40ad-835c-0df126ea4729" (UID: "5624412f-4ebe-40ad-835c-0df126ea4729"). InnerVolumeSpecName "kube-api-access-d2jwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:44.727264 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:44.727175 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5624412f-4ebe-40ad-835c-0df126ea4729-maas-api-xks-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:44.727264 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:44.727207 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2jwr\" (UniqueName: \"kubernetes.io/projected/5624412f-4ebe-40ad-835c-0df126ea4729-kube-api-access-d2jwr\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:44.727264 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:44.727222 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5624412f-4ebe-40ad-835c-0df126ea4729-maas-api-openshift-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:45.391328 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:45.391297 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z424d" Nov 03 00:31:45.391328 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:45.391319 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z424d" event={"ID":"5624412f-4ebe-40ad-835c-0df126ea4729","Type":"ContainerDied","Data":"805f3940eced0e7b5e8c8e3bd41612fbd2d5fecf9059ea295bbde30fc9401b60"} Nov 03 00:31:45.391556 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:45.391363 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="805f3940eced0e7b5e8c8e3bd41612fbd2d5fecf9059ea295bbde30fc9401b60" Nov 03 00:31:53.681275 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:53.681241 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bk6rr"] Nov 03 00:31:53.681750 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:53.681736 5727 cpu_manager.go:414] "containerMap: removing container" podUID="5624412f-4ebe-40ad-835c-0df126ea4729" containerName="revoke-keys" Nov 03 00:31:53.681798 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:53.681752 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5624412f-4ebe-40ad-835c-0df126ea4729" containerName="revoke-keys" Nov 03 00:31:53.681863 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:53.681854 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="5624412f-4ebe-40ad-835c-0df126ea4729" containerName="revoke-keys" Nov 03 00:31:53.685450 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:53.685425 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bk6rr" Nov 03 00:31:53.742683 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:53.741835 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bk6rr"] Nov 03 00:31:53.787476 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:53.787443 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:31:53.787644 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:53.787472 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:31:53.787644 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:53.787500 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:31:53.787721 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:53.787663 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:31:53.815829 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:53.815798 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80cd4ece-bcb6-4c3f-a303-b5d3100d638c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-bk6rr\" (UID: \"80cd4ece-bcb6-4c3f-a303-b5d3100d638c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bk6rr" Nov 03 00:31:53.815982 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:53.815835 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxtvl\" (UniqueName: \"kubernetes.io/projected/80cd4ece-bcb6-4c3f-a303-b5d3100d638c-kube-api-access-mxtvl\") pod \"maas-api-revoke-sub-ec843018-bk6rr\" (UID: \"80cd4ece-bcb6-4c3f-a303-b5d3100d638c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bk6rr" Nov 03 00:31:53.815982 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:53.815869 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80cd4ece-bcb6-4c3f-a303-b5d3100d638c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-bk6rr\" (UID: \"80cd4ece-bcb6-4c3f-a303-b5d3100d638c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bk6rr" Nov 03 00:31:53.916557 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:53.916528 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80cd4ece-bcb6-4c3f-a303-b5d3100d638c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-bk6rr\" (UID: \"80cd4ece-bcb6-4c3f-a303-b5d3100d638c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bk6rr" Nov 03 00:31:53.916708 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:53.916611 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80cd4ece-bcb6-4c3f-a303-b5d3100d638c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-bk6rr\" (UID: \"80cd4ece-bcb6-4c3f-a303-b5d3100d638c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bk6rr" Nov 03 00:31:53.916708 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:53.916636 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxtvl\" (UniqueName: \"kubernetes.io/projected/80cd4ece-bcb6-4c3f-a303-b5d3100d638c-kube-api-access-mxtvl\") pod \"maas-api-revoke-sub-ec843018-bk6rr\" (UID: \"80cd4ece-bcb6-4c3f-a303-b5d3100d638c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bk6rr" Nov 03 00:31:53.917186 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:53.917121 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80cd4ece-bcb6-4c3f-a303-b5d3100d638c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-bk6rr\" (UID: \"80cd4ece-bcb6-4c3f-a303-b5d3100d638c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bk6rr" Nov 03 00:31:53.919097 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:53.919071 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80cd4ece-bcb6-4c3f-a303-b5d3100d638c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-bk6rr\" (UID: \"80cd4ece-bcb6-4c3f-a303-b5d3100d638c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bk6rr" Nov 03 00:31:53.977065 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:53.976999 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxtvl\" (UniqueName: \"kubernetes.io/projected/80cd4ece-bcb6-4c3f-a303-b5d3100d638c-kube-api-access-mxtvl\") pod \"maas-api-revoke-sub-ec843018-bk6rr\" (UID: \"80cd4ece-bcb6-4c3f-a303-b5d3100d638c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bk6rr" Nov 03 00:31:53.999372 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:53.999349 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bk6rr" Nov 03 00:31:54.373818 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:54.373783 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bk6rr"] Nov 03 00:31:54.375559 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:31:54.375534 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80cd4ece_bcb6_4c3f_a303_b5d3100d638c.slice/crio-62a3d606a46d925a62038b80bf56fb210bf50eba6e077f76609c5605930598a0 WatchSource:0}: Error finding container 62a3d606a46d925a62038b80bf56fb210bf50eba6e077f76609c5605930598a0: Status 404 returned error can't find the container with id 62a3d606a46d925a62038b80bf56fb210bf50eba6e077f76609c5605930598a0 Nov 03 00:31:54.452178 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:54.452145 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bk6rr" event={"ID":"80cd4ece-bcb6-4c3f-a303-b5d3100d638c","Type":"ContainerStarted","Data":"62a3d606a46d925a62038b80bf56fb210bf50eba6e077f76609c5605930598a0"} Nov 03 00:31:55.460263 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:55.460229 5727 generic.go:356] "Generic (PLEG): container finished" podID="80cd4ece-bcb6-4c3f-a303-b5d3100d638c" containerID="d459a28d6af98e395de959e5b0b5a2df389c900bf5e1d13cc14b810c804c8635" exitCode=0 Nov 03 00:31:55.460617 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:55.460323 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bk6rr" event={"ID":"80cd4ece-bcb6-4c3f-a303-b5d3100d638c","Type":"ContainerDied","Data":"d459a28d6af98e395de959e5b0b5a2df389c900bf5e1d13cc14b810c804c8635"} Nov 03 00:31:56.623294 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:56.623265 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bk6rr" Nov 03 00:31:56.740863 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:56.740781 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80cd4ece-bcb6-4c3f-a303-b5d3100d638c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80cd4ece-bcb6-4c3f-a303-b5d3100d638c-maas-api-xks-ca\") pod \"80cd4ece-bcb6-4c3f-a303-b5d3100d638c\" (UID: \"80cd4ece-bcb6-4c3f-a303-b5d3100d638c\") " Nov 03 00:31:56.740863 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:56.740832 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80cd4ece-bcb6-4c3f-a303-b5d3100d638c-kube-api-access-mxtvl\" (UniqueName: \"kubernetes.io/projected/80cd4ece-bcb6-4c3f-a303-b5d3100d638c-kube-api-access-mxtvl\") pod \"80cd4ece-bcb6-4c3f-a303-b5d3100d638c\" (UID: \"80cd4ece-bcb6-4c3f-a303-b5d3100d638c\") " Nov 03 00:31:56.741931 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:56.741009 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80cd4ece-bcb6-4c3f-a303-b5d3100d638c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80cd4ece-bcb6-4c3f-a303-b5d3100d638c-maas-api-openshift-ca\") pod \"80cd4ece-bcb6-4c3f-a303-b5d3100d638c\" (UID: \"80cd4ece-bcb6-4c3f-a303-b5d3100d638c\") " Nov 03 00:31:56.741931 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:56.741701 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80cd4ece-bcb6-4c3f-a303-b5d3100d638c-maas-api-openshift-ca" pod "80cd4ece-bcb6-4c3f-a303-b5d3100d638c" (UID: "80cd4ece-bcb6-4c3f-a303-b5d3100d638c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:31:56.744605 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:56.744581 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80cd4ece-bcb6-4c3f-a303-b5d3100d638c-maas-api-xks-ca" pod "80cd4ece-bcb6-4c3f-a303-b5d3100d638c" (UID: "80cd4ece-bcb6-4c3f-a303-b5d3100d638c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:31:56.744742 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:56.744705 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80cd4ece-bcb6-4c3f-a303-b5d3100d638c-kube-api-access-mxtvl" pod "80cd4ece-bcb6-4c3f-a303-b5d3100d638c" (UID: "80cd4ece-bcb6-4c3f-a303-b5d3100d638c"). InnerVolumeSpecName "kube-api-access-mxtvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:56.842379 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:56.842348 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80cd4ece-bcb6-4c3f-a303-b5d3100d638c-maas-api-openshift-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:56.842379 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:56.842378 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80cd4ece-bcb6-4c3f-a303-b5d3100d638c-maas-api-xks-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:56.842594 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:56.842390 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxtvl\" (UniqueName: \"kubernetes.io/projected/80cd4ece-bcb6-4c3f-a303-b5d3100d638c-kube-api-access-mxtvl\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:57.474427 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:57.474344 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bk6rr" Nov 03 00:31:57.474427 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:57.474369 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bk6rr" event={"ID":"80cd4ece-bcb6-4c3f-a303-b5d3100d638c","Type":"ContainerDied","Data":"62a3d606a46d925a62038b80bf56fb210bf50eba6e077f76609c5605930598a0"} Nov 03 00:31:57.474427 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:31:57.474421 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62a3d606a46d925a62038b80bf56fb210bf50eba6e077f76609c5605930598a0" Nov 03 00:31:57.845097 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:31:57.845059 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:32:01.051760 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:01.051727 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sj86z"] Nov 03 00:32:01.101103 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:01.101065 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sj86z"] Nov 03 00:32:01.882526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:01.882494 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9f5f6db-e1de-4901-b724-b95417c3ff6a" path="/var/lib/kubelet/pods/a9f5f6db-e1de-4901-b724-b95417c3ff6a/volumes" Nov 03 00:32:27.844578 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:32:27.844541 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:32:28.964275 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:28.964237 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7rqm7"] Nov 03 00:32:28.964780 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:28.964707 5727 cpu_manager.go:414] "containerMap: removing container" podUID="80cd4ece-bcb6-4c3f-a303-b5d3100d638c" containerName="revoke-keys" Nov 03 00:32:28.964780 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:28.964717 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80cd4ece-bcb6-4c3f-a303-b5d3100d638c" containerName="revoke-keys" Nov 03 00:32:28.964921 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:28.964795 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="80cd4ece-bcb6-4c3f-a303-b5d3100d638c" containerName="revoke-keys" Nov 03 00:32:28.968154 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:28.968119 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7rqm7" Nov 03 00:32:29.022192 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:29.022167 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:32:29.058613 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:29.058583 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c1586c0-c0e3-47e3-a0c4-132e66b50514-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-7rqm7\" (UID: \"7c1586c0-c0e3-47e3-a0c4-132e66b50514\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7rqm7" Nov 03 00:32:29.058780 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:29.058646 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlzzn\" (UniqueName: \"kubernetes.io/projected/7c1586c0-c0e3-47e3-a0c4-132e66b50514-kube-api-access-nlzzn\") pod \"maas-api-revoke-sub-046e1343-7rqm7\" (UID: \"7c1586c0-c0e3-47e3-a0c4-132e66b50514\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7rqm7" Nov 03 00:32:29.058857 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:29.058798 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c1586c0-c0e3-47e3-a0c4-132e66b50514-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-7rqm7\" (UID: \"7c1586c0-c0e3-47e3-a0c4-132e66b50514\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7rqm7" Nov 03 00:32:29.071684 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:29.071653 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:32:29.071684 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:29.071664 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:32:29.071852 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:29.071712 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:32:29.074733 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:29.074712 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7rqm7"] Nov 03 00:32:29.159377 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:29.159350 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlzzn\" (UniqueName: \"kubernetes.io/projected/7c1586c0-c0e3-47e3-a0c4-132e66b50514-kube-api-access-nlzzn\") pod \"maas-api-revoke-sub-046e1343-7rqm7\" (UID: \"7c1586c0-c0e3-47e3-a0c4-132e66b50514\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7rqm7" Nov 03 00:32:29.159550 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:29.159414 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c1586c0-c0e3-47e3-a0c4-132e66b50514-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-7rqm7\" (UID: \"7c1586c0-c0e3-47e3-a0c4-132e66b50514\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7rqm7" Nov 03 00:32:29.159550 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:29.159478 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c1586c0-c0e3-47e3-a0c4-132e66b50514-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-7rqm7\" (UID: \"7c1586c0-c0e3-47e3-a0c4-132e66b50514\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7rqm7" Nov 03 00:32:29.160185 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:29.160158 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c1586c0-c0e3-47e3-a0c4-132e66b50514-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-7rqm7\" (UID: \"7c1586c0-c0e3-47e3-a0c4-132e66b50514\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7rqm7" Nov 03 00:32:29.162036 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:29.162017 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c1586c0-c0e3-47e3-a0c4-132e66b50514-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-7rqm7\" (UID: \"7c1586c0-c0e3-47e3-a0c4-132e66b50514\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7rqm7" Nov 03 00:32:29.219884 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:29.219830 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlzzn\" (UniqueName: \"kubernetes.io/projected/7c1586c0-c0e3-47e3-a0c4-132e66b50514-kube-api-access-nlzzn\") pod \"maas-api-revoke-sub-046e1343-7rqm7\" (UID: \"7c1586c0-c0e3-47e3-a0c4-132e66b50514\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7rqm7" Nov 03 00:32:29.281863 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:29.281841 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7rqm7" Nov 03 00:32:29.657034 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:29.657009 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7rqm7"] Nov 03 00:32:29.658934 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:32:29.658907 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c1586c0_c0e3_47e3_a0c4_132e66b50514.slice/crio-50c4fea13ced195c3d12383865e5698b088824387ed1d71d806215f8c66ff8d7 WatchSource:0}: Error finding container 50c4fea13ced195c3d12383865e5698b088824387ed1d71d806215f8c66ff8d7: Status 404 returned error can't find the container with id 50c4fea13ced195c3d12383865e5698b088824387ed1d71d806215f8c66ff8d7 Nov 03 00:32:29.720456 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:29.720428 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7rqm7" event={"ID":"7c1586c0-c0e3-47e3-a0c4-132e66b50514","Type":"ContainerStarted","Data":"50c4fea13ced195c3d12383865e5698b088824387ed1d71d806215f8c66ff8d7"} Nov 03 00:32:30.736411 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:30.736310 5727 generic.go:356] "Generic (PLEG): container finished" podID="7c1586c0-c0e3-47e3-a0c4-132e66b50514" containerID="afe9f65098b86fc6762c5e65fac00a00747b445e71b018d4a4a8b82d774c8966" exitCode=0 Nov 03 00:32:30.736868 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:30.736475 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7rqm7" event={"ID":"7c1586c0-c0e3-47e3-a0c4-132e66b50514","Type":"ContainerDied","Data":"afe9f65098b86fc6762c5e65fac00a00747b445e71b018d4a4a8b82d774c8966"} Nov 03 00:32:31.887220 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:31.887196 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7rqm7" Nov 03 00:32:31.987041 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:31.987008 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c1586c0-c0e3-47e3-a0c4-132e66b50514-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c1586c0-c0e3-47e3-a0c4-132e66b50514-maas-api-xks-ca\") pod \"7c1586c0-c0e3-47e3-a0c4-132e66b50514\" (UID: \"7c1586c0-c0e3-47e3-a0c4-132e66b50514\") " Nov 03 00:32:31.987235 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:31.987063 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c1586c0-c0e3-47e3-a0c4-132e66b50514-kube-api-access-nlzzn\" (UniqueName: \"kubernetes.io/projected/7c1586c0-c0e3-47e3-a0c4-132e66b50514-kube-api-access-nlzzn\") pod \"7c1586c0-c0e3-47e3-a0c4-132e66b50514\" (UID: \"7c1586c0-c0e3-47e3-a0c4-132e66b50514\") " Nov 03 00:32:31.987235 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:31.987092 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c1586c0-c0e3-47e3-a0c4-132e66b50514-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c1586c0-c0e3-47e3-a0c4-132e66b50514-maas-api-openshift-ca\") pod \"7c1586c0-c0e3-47e3-a0c4-132e66b50514\" (UID: \"7c1586c0-c0e3-47e3-a0c4-132e66b50514\") " Nov 03 00:32:31.987548 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:31.987519 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c1586c0-c0e3-47e3-a0c4-132e66b50514-maas-api-openshift-ca" pod "7c1586c0-c0e3-47e3-a0c4-132e66b50514" (UID: "7c1586c0-c0e3-47e3-a0c4-132e66b50514"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:32:31.989366 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:31.989341 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c1586c0-c0e3-47e3-a0c4-132e66b50514-maas-api-xks-ca" pod "7c1586c0-c0e3-47e3-a0c4-132e66b50514" (UID: "7c1586c0-c0e3-47e3-a0c4-132e66b50514"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:32:31.989473 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:31.989384 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c1586c0-c0e3-47e3-a0c4-132e66b50514-kube-api-access-nlzzn" pod "7c1586c0-c0e3-47e3-a0c4-132e66b50514" (UID: "7c1586c0-c0e3-47e3-a0c4-132e66b50514"). InnerVolumeSpecName "kube-api-access-nlzzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:32:32.088921 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:32.088889 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c1586c0-c0e3-47e3-a0c4-132e66b50514-maas-api-xks-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:32:32.088921 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:32.088917 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlzzn\" (UniqueName: \"kubernetes.io/projected/7c1586c0-c0e3-47e3-a0c4-132e66b50514-kube-api-access-nlzzn\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:32:32.088921 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:32.088927 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c1586c0-c0e3-47e3-a0c4-132e66b50514-maas-api-openshift-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:32:32.750888 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:32.750859 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7rqm7" Nov 03 00:32:32.751083 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:32.750900 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7rqm7" event={"ID":"7c1586c0-c0e3-47e3-a0c4-132e66b50514","Type":"ContainerDied","Data":"50c4fea13ced195c3d12383865e5698b088824387ed1d71d806215f8c66ff8d7"} Nov 03 00:32:32.751083 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:32.750948 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50c4fea13ced195c3d12383865e5698b088824387ed1d71d806215f8c66ff8d7" Nov 03 00:32:35.046998 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:35.046966 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pnj52"] Nov 03 00:32:35.097392 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:35.097361 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pnj52"] Nov 03 00:32:35.882033 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:35.881998 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="216fa6c9-c0de-4286-abbe-314fb439bb62" path="/var/lib/kubelet/pods/216fa6c9-c0de-4286-abbe-314fb439bb62/volumes" Nov 03 00:32:39.759150 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:39.759092 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtppm"] Nov 03 00:32:39.759760 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:39.759741 5727 cpu_manager.go:414] "containerMap: removing container" podUID="7c1586c0-c0e3-47e3-a0c4-132e66b50514" containerName="revoke-keys" Nov 03 00:32:39.759760 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:39.759761 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c1586c0-c0e3-47e3-a0c4-132e66b50514" containerName="revoke-keys" Nov 03 00:32:39.759904 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:39.759882 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c1586c0-c0e3-47e3-a0c4-132e66b50514" containerName="revoke-keys" Nov 03 00:32:39.764141 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:39.764109 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtppm" Nov 03 00:32:39.860720 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:39.860685 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjsq9\" (UniqueName: \"kubernetes.io/projected/23db4f9f-8f48-4832-ad67-d9febdfb5633-kube-api-access-fjsq9\") pod \"maas-api-revoke-sub-d3fba056-mtppm\" (UID: \"23db4f9f-8f48-4832-ad67-d9febdfb5633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtppm" Nov 03 00:32:39.860720 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:39.860723 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23db4f9f-8f48-4832-ad67-d9febdfb5633-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-mtppm\" (UID: \"23db4f9f-8f48-4832-ad67-d9febdfb5633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtppm" Nov 03 00:32:39.860929 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:39.860743 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23db4f9f-8f48-4832-ad67-d9febdfb5633-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-mtppm\" (UID: \"23db4f9f-8f48-4832-ad67-d9febdfb5633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtppm" Nov 03 00:32:39.866613 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:39.866585 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:32:39.866613 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:39.866591 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:32:39.866613 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:39.866606 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:32:39.866840 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:39.866665 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:32:39.869376 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:39.869358 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtppm"] Nov 03 00:32:39.961632 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:39.961597 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjsq9\" (UniqueName: \"kubernetes.io/projected/23db4f9f-8f48-4832-ad67-d9febdfb5633-kube-api-access-fjsq9\") pod \"maas-api-revoke-sub-d3fba056-mtppm\" (UID: \"23db4f9f-8f48-4832-ad67-d9febdfb5633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtppm" Nov 03 00:32:39.961632 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:39.961632 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23db4f9f-8f48-4832-ad67-d9febdfb5633-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-mtppm\" (UID: \"23db4f9f-8f48-4832-ad67-d9febdfb5633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtppm" Nov 03 00:32:39.961889 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:39.961653 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23db4f9f-8f48-4832-ad67-d9febdfb5633-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-mtppm\" (UID: \"23db4f9f-8f48-4832-ad67-d9febdfb5633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtppm" Nov 03 00:32:39.962379 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:39.962350 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23db4f9f-8f48-4832-ad67-d9febdfb5633-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-mtppm\" (UID: \"23db4f9f-8f48-4832-ad67-d9febdfb5633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtppm" Nov 03 00:32:39.964247 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:39.964226 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23db4f9f-8f48-4832-ad67-d9febdfb5633-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-mtppm\" (UID: \"23db4f9f-8f48-4832-ad67-d9febdfb5633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtppm" Nov 03 00:32:40.022627 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:40.022547 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjsq9\" (UniqueName: \"kubernetes.io/projected/23db4f9f-8f48-4832-ad67-d9febdfb5633-kube-api-access-fjsq9\") pod \"maas-api-revoke-sub-d3fba056-mtppm\" (UID: \"23db4f9f-8f48-4832-ad67-d9febdfb5633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtppm" Nov 03 00:32:40.076618 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:40.076584 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtppm" Nov 03 00:32:40.457492 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:40.457467 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtppm"] Nov 03 00:32:40.459655 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:32:40.459624 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23db4f9f_8f48_4832_ad67_d9febdfb5633.slice/crio-0c63ed854f0617f928aa221b8c1b6b9d2dd19b3649beeadb9a091a4895c1eb3d WatchSource:0}: Error finding container 0c63ed854f0617f928aa221b8c1b6b9d2dd19b3649beeadb9a091a4895c1eb3d: Status 404 returned error can't find the container with id 0c63ed854f0617f928aa221b8c1b6b9d2dd19b3649beeadb9a091a4895c1eb3d Nov 03 00:32:40.806223 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:40.806186 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtppm" event={"ID":"23db4f9f-8f48-4832-ad67-d9febdfb5633","Type":"ContainerStarted","Data":"de021c7877237aeb307b523f9c30a6f216e567e7d981f5226b9d2e9e7195ace6"} Nov 03 00:32:40.806656 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:40.806241 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtppm" event={"ID":"23db4f9f-8f48-4832-ad67-d9febdfb5633","Type":"ContainerStarted","Data":"0c63ed854f0617f928aa221b8c1b6b9d2dd19b3649beeadb9a091a4895c1eb3d"} Nov 03 00:32:40.922506 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:40.922454 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtppm" podStartSLOduration=1.922440895 podStartE2EDuration="1.922440895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:32:40.920598345 +0000 UTC m=+6247.159096009" watchObservedRunningTime="2026-11-03 00:32:40.922440895 +0000 UTC m=+6247.160938556" Nov 03 00:32:41.823366 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:41.823334 5727 generic.go:356] "Generic (PLEG): container finished" podID="23db4f9f-8f48-4832-ad67-d9febdfb5633" containerID="de021c7877237aeb307b523f9c30a6f216e567e7d981f5226b9d2e9e7195ace6" exitCode=0 Nov 03 00:32:41.823798 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:41.823382 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtppm" event={"ID":"23db4f9f-8f48-4832-ad67-d9febdfb5633","Type":"ContainerDied","Data":"de021c7877237aeb307b523f9c30a6f216e567e7d981f5226b9d2e9e7195ace6"} Nov 03 00:32:43.000230 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:43.000193 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtppm" Nov 03 00:32:43.102216 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:43.101607 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23db4f9f-8f48-4832-ad67-d9febdfb5633-kube-api-access-fjsq9\" (UniqueName: \"kubernetes.io/projected/23db4f9f-8f48-4832-ad67-d9febdfb5633-kube-api-access-fjsq9\") pod \"23db4f9f-8f48-4832-ad67-d9febdfb5633\" (UID: \"23db4f9f-8f48-4832-ad67-d9febdfb5633\") " Nov 03 00:32:43.102216 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:43.101716 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23db4f9f-8f48-4832-ad67-d9febdfb5633-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23db4f9f-8f48-4832-ad67-d9febdfb5633-maas-api-xks-ca\") pod \"23db4f9f-8f48-4832-ad67-d9febdfb5633\" (UID: \"23db4f9f-8f48-4832-ad67-d9febdfb5633\") " Nov 03 00:32:43.102216 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:43.101756 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23db4f9f-8f48-4832-ad67-d9febdfb5633-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23db4f9f-8f48-4832-ad67-d9febdfb5633-maas-api-openshift-ca\") pod \"23db4f9f-8f48-4832-ad67-d9febdfb5633\" (UID: \"23db4f9f-8f48-4832-ad67-d9febdfb5633\") " Nov 03 00:32:43.103315 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:43.103261 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23db4f9f-8f48-4832-ad67-d9febdfb5633-maas-api-openshift-ca" pod "23db4f9f-8f48-4832-ad67-d9febdfb5633" (UID: "23db4f9f-8f48-4832-ad67-d9febdfb5633"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:32:43.132496 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:43.121262 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23db4f9f-8f48-4832-ad67-d9febdfb5633-maas-api-xks-ca" pod "23db4f9f-8f48-4832-ad67-d9febdfb5633" (UID: "23db4f9f-8f48-4832-ad67-d9febdfb5633"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:32:43.132496 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:43.127598 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23db4f9f-8f48-4832-ad67-d9febdfb5633-kube-api-access-fjsq9" pod "23db4f9f-8f48-4832-ad67-d9febdfb5633" (UID: "23db4f9f-8f48-4832-ad67-d9febdfb5633"). InnerVolumeSpecName "kube-api-access-fjsq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:32:43.208153 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:43.205991 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjsq9\" (UniqueName: \"kubernetes.io/projected/23db4f9f-8f48-4832-ad67-d9febdfb5633-kube-api-access-fjsq9\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:32:43.208153 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:43.206045 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23db4f9f-8f48-4832-ad67-d9febdfb5633-maas-api-xks-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:32:43.208153 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:43.206064 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23db4f9f-8f48-4832-ad67-d9febdfb5633-maas-api-openshift-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:32:43.842872 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:43.842722 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtppm" Nov 03 00:32:43.842872 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:43.842774 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtppm" event={"ID":"23db4f9f-8f48-4832-ad67-d9febdfb5633","Type":"ContainerDied","Data":"0c63ed854f0617f928aa221b8c1b6b9d2dd19b3649beeadb9a091a4895c1eb3d"} Nov 03 00:32:43.842872 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:32:43.842836 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c63ed854f0617f928aa221b8c1b6b9d2dd19b3649beeadb9a091a4895c1eb3d" Nov 03 00:32:57.848321 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:32:57.848205 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:33:00.050464 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:00.050425 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c1fb147-5jrzt"] Nov 03 00:33:00.099995 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:00.099964 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c1fb147-5jrzt"] Nov 03 00:33:01.889059 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:01.889016 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a034249-04ab-4783-9adb-d5ce03de6671" path="/var/lib/kubelet/pods/0a034249-04ab-4783-9adb-d5ce03de6671/volumes" Nov 03 00:33:14.590112 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:14.590074 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hkq8r"] Nov 03 00:33:14.590576 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:14.590563 5727 cpu_manager.go:414] "containerMap: removing container" podUID="23db4f9f-8f48-4832-ad67-d9febdfb5633" containerName="revoke-keys" Nov 03 00:33:14.590623 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:14.590577 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23db4f9f-8f48-4832-ad67-d9febdfb5633" containerName="revoke-keys" Nov 03 00:33:14.590677 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:14.590668 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="23db4f9f-8f48-4832-ad67-d9febdfb5633" containerName="revoke-keys" Nov 03 00:33:14.595260 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:14.595243 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hkq8r" Nov 03 00:33:14.647540 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:14.647490 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:33:14.658964 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:14.658928 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp2c7\" (UniqueName: \"kubernetes.io/projected/053fafe7-6bb7-439b-91c7-f33dc833cae2-kube-api-access-dp2c7\") pod \"maas-api-revoke-sub-ae594979-hkq8r\" (UID: \"053fafe7-6bb7-439b-91c7-f33dc833cae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hkq8r" Nov 03 00:33:14.659182 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:14.658969 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/053fafe7-6bb7-439b-91c7-f33dc833cae2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-hkq8r\" (UID: \"053fafe7-6bb7-439b-91c7-f33dc833cae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hkq8r" Nov 03 00:33:14.659182 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:14.659004 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/053fafe7-6bb7-439b-91c7-f33dc833cae2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-hkq8r\" (UID: \"053fafe7-6bb7-439b-91c7-f33dc833cae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hkq8r" Nov 03 00:33:14.697244 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:14.697207 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:33:14.697456 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:14.697311 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:33:14.697518 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:14.697454 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:33:14.700357 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:14.700332 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hkq8r"] Nov 03 00:33:14.760512 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:14.760470 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp2c7\" (UniqueName: \"kubernetes.io/projected/053fafe7-6bb7-439b-91c7-f33dc833cae2-kube-api-access-dp2c7\") pod \"maas-api-revoke-sub-ae594979-hkq8r\" (UID: \"053fafe7-6bb7-439b-91c7-f33dc833cae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hkq8r" Nov 03 00:33:14.760512 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:14.760513 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/053fafe7-6bb7-439b-91c7-f33dc833cae2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-hkq8r\" (UID: \"053fafe7-6bb7-439b-91c7-f33dc833cae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hkq8r" Nov 03 00:33:14.760774 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:14.760563 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/053fafe7-6bb7-439b-91c7-f33dc833cae2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-hkq8r\" (UID: \"053fafe7-6bb7-439b-91c7-f33dc833cae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hkq8r" Nov 03 00:33:14.761432 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:14.761401 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/053fafe7-6bb7-439b-91c7-f33dc833cae2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-hkq8r\" (UID: \"053fafe7-6bb7-439b-91c7-f33dc833cae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hkq8r" Nov 03 00:33:14.763305 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:14.763282 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/053fafe7-6bb7-439b-91c7-f33dc833cae2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-hkq8r\" (UID: \"053fafe7-6bb7-439b-91c7-f33dc833cae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hkq8r" Nov 03 00:33:14.821148 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:14.821101 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp2c7\" (UniqueName: \"kubernetes.io/projected/053fafe7-6bb7-439b-91c7-f33dc833cae2-kube-api-access-dp2c7\") pod \"maas-api-revoke-sub-ae594979-hkq8r\" (UID: \"053fafe7-6bb7-439b-91c7-f33dc833cae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hkq8r" Nov 03 00:33:14.908584 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:14.908498 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hkq8r" Nov 03 00:33:15.284413 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:15.284386 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hkq8r"] Nov 03 00:33:15.286044 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:33:15.286017 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod053fafe7_6bb7_439b_91c7_f33dc833cae2.slice/crio-6030b93d1468697dbb88a0144da39be581e9be1b0d78a7ed432b466644b36c4c WatchSource:0}: Error finding container 6030b93d1468697dbb88a0144da39be581e9be1b0d78a7ed432b466644b36c4c: Status 404 returned error can't find the container with id 6030b93d1468697dbb88a0144da39be581e9be1b0d78a7ed432b466644b36c4c Nov 03 00:33:16.050465 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:16.050424 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hkq8r" event={"ID":"053fafe7-6bb7-439b-91c7-f33dc833cae2","Type":"ContainerStarted","Data":"a1bd0ba593557524faa7fa6d1ba965f02b65a181e3b51f62059687f713947a1c"} Nov 03 00:33:16.050883 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:16.050489 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hkq8r" event={"ID":"053fafe7-6bb7-439b-91c7-f33dc833cae2","Type":"ContainerStarted","Data":"6030b93d1468697dbb88a0144da39be581e9be1b0d78a7ed432b466644b36c4c"} Nov 03 00:33:16.165966 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:16.165918 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hkq8r" podStartSLOduration=2.165904854 podStartE2EDuration="2.165904854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:16.165275196 +0000 UTC m=+6282.403772862" watchObservedRunningTime="2026-11-03 00:33:16.165904854 +0000 UTC m=+6282.404402515" Nov 03 00:33:17.057847 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:17.057812 5727 generic.go:356] "Generic (PLEG): container finished" podID="053fafe7-6bb7-439b-91c7-f33dc833cae2" containerID="a1bd0ba593557524faa7fa6d1ba965f02b65a181e3b51f62059687f713947a1c" exitCode=0 Nov 03 00:33:17.058277 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:17.057864 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hkq8r" event={"ID":"053fafe7-6bb7-439b-91c7-f33dc833cae2","Type":"ContainerDied","Data":"a1bd0ba593557524faa7fa6d1ba965f02b65a181e3b51f62059687f713947a1c"} Nov 03 00:33:18.207026 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:18.207002 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hkq8r" Nov 03 00:33:18.294295 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:18.294248 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/053fafe7-6bb7-439b-91c7-f33dc833cae2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/053fafe7-6bb7-439b-91c7-f33dc833cae2-maas-api-xks-ca\") pod \"053fafe7-6bb7-439b-91c7-f33dc833cae2\" (UID: \"053fafe7-6bb7-439b-91c7-f33dc833cae2\") " Nov 03 00:33:18.294475 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:18.294372 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/053fafe7-6bb7-439b-91c7-f33dc833cae2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/053fafe7-6bb7-439b-91c7-f33dc833cae2-maas-api-openshift-ca\") pod \"053fafe7-6bb7-439b-91c7-f33dc833cae2\" (UID: \"053fafe7-6bb7-439b-91c7-f33dc833cae2\") " Nov 03 00:33:18.294475 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:18.294417 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/053fafe7-6bb7-439b-91c7-f33dc833cae2-kube-api-access-dp2c7\" (UniqueName: \"kubernetes.io/projected/053fafe7-6bb7-439b-91c7-f33dc833cae2-kube-api-access-dp2c7\") pod \"053fafe7-6bb7-439b-91c7-f33dc833cae2\" (UID: \"053fafe7-6bb7-439b-91c7-f33dc833cae2\") " Nov 03 00:33:18.294721 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:18.294698 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/053fafe7-6bb7-439b-91c7-f33dc833cae2-maas-api-openshift-ca" pod "053fafe7-6bb7-439b-91c7-f33dc833cae2" (UID: "053fafe7-6bb7-439b-91c7-f33dc833cae2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:33:18.296592 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:18.296564 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/053fafe7-6bb7-439b-91c7-f33dc833cae2-maas-api-xks-ca" pod "053fafe7-6bb7-439b-91c7-f33dc833cae2" (UID: "053fafe7-6bb7-439b-91c7-f33dc833cae2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:33:18.296711 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:18.296651 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/053fafe7-6bb7-439b-91c7-f33dc833cae2-kube-api-access-dp2c7" pod "053fafe7-6bb7-439b-91c7-f33dc833cae2" (UID: "053fafe7-6bb7-439b-91c7-f33dc833cae2"). InnerVolumeSpecName "kube-api-access-dp2c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:33:18.395858 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:18.395780 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/053fafe7-6bb7-439b-91c7-f33dc833cae2-maas-api-xks-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:18.395858 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:18.395809 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/053fafe7-6bb7-439b-91c7-f33dc833cae2-maas-api-openshift-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:18.395858 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:18.395820 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp2c7\" (UniqueName: \"kubernetes.io/projected/053fafe7-6bb7-439b-91c7-f33dc833cae2-kube-api-access-dp2c7\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:19.071854 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:19.071818 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hkq8r" Nov 03 00:33:19.072043 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:19.071857 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hkq8r" event={"ID":"053fafe7-6bb7-439b-91c7-f33dc833cae2","Type":"ContainerDied","Data":"6030b93d1468697dbb88a0144da39be581e9be1b0d78a7ed432b466644b36c4c"} Nov 03 00:33:19.072043 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:19.071907 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6030b93d1468697dbb88a0144da39be581e9be1b0d78a7ed432b466644b36c4c" Nov 03 00:33:27.845467 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:33:27.845428 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:33:33.394890 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:33.394852 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2cjz"] Nov 03 00:33:33.421208 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:33.414541 5727 cpu_manager.go:414] "containerMap: removing container" podUID="053fafe7-6bb7-439b-91c7-f33dc833cae2" containerName="revoke-keys" Nov 03 00:33:33.421208 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:33.414577 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="053fafe7-6bb7-439b-91c7-f33dc833cae2" containerName="revoke-keys" Nov 03 00:33:33.421208 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:33.414775 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="053fafe7-6bb7-439b-91c7-f33dc833cae2" containerName="revoke-keys" Nov 03 00:33:33.422455 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:33.422430 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2cjz" Nov 03 00:33:33.442516 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:33.442487 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2cjz"] Nov 03 00:33:33.513154 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:33.488403 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:33:33.513422 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:33.489840 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:33:33.523154 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:33.489910 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:33:33.538805 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:33.538767 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:33:33.638993 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:33.638960 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcppt\" (UniqueName: \"kubernetes.io/projected/c84aa5c3-904d-4577-99c3-48886c07a617-kube-api-access-qcppt\") pod \"maas-api-revoke-sub-a4aef7d7-j2cjz\" (UID: \"c84aa5c3-904d-4577-99c3-48886c07a617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2cjz" Nov 03 00:33:33.643307 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:33.643278 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84aa5c3-904d-4577-99c3-48886c07a617-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-j2cjz\" (UID: \"c84aa5c3-904d-4577-99c3-48886c07a617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2cjz" Nov 03 00:33:33.647439 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:33.647361 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84aa5c3-904d-4577-99c3-48886c07a617-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-j2cjz\" (UID: \"c84aa5c3-904d-4577-99c3-48886c07a617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2cjz" Nov 03 00:33:33.748728 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:33.748682 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84aa5c3-904d-4577-99c3-48886c07a617-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-j2cjz\" (UID: \"c84aa5c3-904d-4577-99c3-48886c07a617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2cjz" Nov 03 00:33:33.748948 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:33.748759 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcppt\" (UniqueName: \"kubernetes.io/projected/c84aa5c3-904d-4577-99c3-48886c07a617-kube-api-access-qcppt\") pod \"maas-api-revoke-sub-a4aef7d7-j2cjz\" (UID: \"c84aa5c3-904d-4577-99c3-48886c07a617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2cjz" Nov 03 00:33:33.748948 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:33.748782 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84aa5c3-904d-4577-99c3-48886c07a617-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-j2cjz\" (UID: \"c84aa5c3-904d-4577-99c3-48886c07a617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2cjz" Nov 03 00:33:33.749414 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:33.749386 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84aa5c3-904d-4577-99c3-48886c07a617-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-j2cjz\" (UID: \"c84aa5c3-904d-4577-99c3-48886c07a617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2cjz" Nov 03 00:33:33.770384 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:33.766908 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84aa5c3-904d-4577-99c3-48886c07a617-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-j2cjz\" (UID: \"c84aa5c3-904d-4577-99c3-48886c07a617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2cjz" Nov 03 00:33:33.815241 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:33.815205 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcppt\" (UniqueName: \"kubernetes.io/projected/c84aa5c3-904d-4577-99c3-48886c07a617-kube-api-access-qcppt\") pod \"maas-api-revoke-sub-a4aef7d7-j2cjz\" (UID: \"c84aa5c3-904d-4577-99c3-48886c07a617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2cjz" Nov 03 00:33:34.097152 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:34.096407 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:33:34.097152 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:34.096480 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2cjz" Nov 03 00:33:34.629427 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:34.629370 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2cjz"] Nov 03 00:33:34.631493 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:33:34.631439 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc84aa5c3_904d_4577_99c3_48886c07a617.slice/crio-0c4b7585305ba73e1f409b45de573ca1e19a4f49b146d35f60682946cfe42581 WatchSource:0}: Error finding container 0c4b7585305ba73e1f409b45de573ca1e19a4f49b146d35f60682946cfe42581: Status 404 returned error can't find the container with id 0c4b7585305ba73e1f409b45de573ca1e19a4f49b146d35f60682946cfe42581 Nov 03 00:33:35.190994 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:35.190897 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2cjz" event={"ID":"c84aa5c3-904d-4577-99c3-48886c07a617","Type":"ContainerStarted","Data":"c98a40a4434f30b35f90ca3e6b909e2bcdac343bbfd75853e212aaa81b177158"} Nov 03 00:33:35.191218 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:35.190997 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2cjz" event={"ID":"c84aa5c3-904d-4577-99c3-48886c07a617","Type":"ContainerStarted","Data":"0c4b7585305ba73e1f409b45de573ca1e19a4f49b146d35f60682946cfe42581"} Nov 03 00:33:35.212699 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:35.212672 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 03 00:33:35.228500 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:35.228479 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 03 00:33:35.307336 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:35.307285 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2cjz" podStartSLOduration=2.307271685 podStartE2EDuration="2.307271685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:35.305470631 +0000 UTC m=+6301.543968295" watchObservedRunningTime="2026-11-03 00:33:35.307271685 +0000 UTC m=+6301.545769388" Nov 03 00:33:35.328730 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:35.328702 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 03 00:33:35.345484 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:35.345460 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 03 00:33:36.199920 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:36.199884 5727 generic.go:356] "Generic (PLEG): container finished" podID="c84aa5c3-904d-4577-99c3-48886c07a617" containerID="c98a40a4434f30b35f90ca3e6b909e2bcdac343bbfd75853e212aaa81b177158" exitCode=0 Nov 03 00:33:36.200364 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:36.199968 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2cjz" event={"ID":"c84aa5c3-904d-4577-99c3-48886c07a617","Type":"ContainerDied","Data":"c98a40a4434f30b35f90ca3e6b909e2bcdac343bbfd75853e212aaa81b177158"} Nov 03 00:33:36.367782 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:36.367741 5727 scope.go:122] "RemoveContainer" containerID="d50fb81d0441768e8bfda3c9f2787be5cdc6d6d6ea849851f419067c18ee7103" Nov 03 00:33:36.385715 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:36.385674 5727 scope.go:122] "RemoveContainer" containerID="25cfd900dceec3a7123262a1fd73ea1cd95416b443bb4201b3dd29bd99d4d937" Nov 03 00:33:37.351780 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:37.351753 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2cjz" Nov 03 00:33:37.453653 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:37.453620 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c84aa5c3-904d-4577-99c3-48886c07a617-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84aa5c3-904d-4577-99c3-48886c07a617-maas-api-xks-ca\") pod \"c84aa5c3-904d-4577-99c3-48886c07a617\" (UID: \"c84aa5c3-904d-4577-99c3-48886c07a617\") " Nov 03 00:33:37.453854 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:37.453665 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c84aa5c3-904d-4577-99c3-48886c07a617-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84aa5c3-904d-4577-99c3-48886c07a617-maas-api-openshift-ca\") pod \"c84aa5c3-904d-4577-99c3-48886c07a617\" (UID: \"c84aa5c3-904d-4577-99c3-48886c07a617\") " Nov 03 00:33:37.453854 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:37.453799 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c84aa5c3-904d-4577-99c3-48886c07a617-kube-api-access-qcppt\" (UniqueName: \"kubernetes.io/projected/c84aa5c3-904d-4577-99c3-48886c07a617-kube-api-access-qcppt\") pod \"c84aa5c3-904d-4577-99c3-48886c07a617\" (UID: \"c84aa5c3-904d-4577-99c3-48886c07a617\") " Nov 03 00:33:37.454038 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:37.454005 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c84aa5c3-904d-4577-99c3-48886c07a617-maas-api-openshift-ca" pod "c84aa5c3-904d-4577-99c3-48886c07a617" (UID: "c84aa5c3-904d-4577-99c3-48886c07a617"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:33:37.454239 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:37.454222 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84aa5c3-904d-4577-99c3-48886c07a617-maas-api-openshift-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:37.456049 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:37.456029 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c84aa5c3-904d-4577-99c3-48886c07a617-maas-api-xks-ca" pod "c84aa5c3-904d-4577-99c3-48886c07a617" (UID: "c84aa5c3-904d-4577-99c3-48886c07a617"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:33:37.456192 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:37.456090 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c84aa5c3-904d-4577-99c3-48886c07a617-kube-api-access-qcppt" pod "c84aa5c3-904d-4577-99c3-48886c07a617" (UID: "c84aa5c3-904d-4577-99c3-48886c07a617"). InnerVolumeSpecName "kube-api-access-qcppt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:33:37.555039 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:37.555007 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcppt\" (UniqueName: \"kubernetes.io/projected/c84aa5c3-904d-4577-99c3-48886c07a617-kube-api-access-qcppt\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:37.555039 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:37.555038 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84aa5c3-904d-4577-99c3-48886c07a617-maas-api-xks-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:38.213778 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:38.213699 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2cjz" Nov 03 00:33:38.213778 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:38.213723 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2cjz" event={"ID":"c84aa5c3-904d-4577-99c3-48886c07a617","Type":"ContainerDied","Data":"0c4b7585305ba73e1f409b45de573ca1e19a4f49b146d35f60682946cfe42581"} Nov 03 00:33:38.213778 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:38.213770 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c4b7585305ba73e1f409b45de573ca1e19a4f49b146d35f60682946cfe42581" Nov 03 00:33:54.535561 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:54.535477 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v2ck7"] Nov 03 00:33:54.535967 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:54.535926 5727 cpu_manager.go:414] "containerMap: removing container" podUID="c84aa5c3-904d-4577-99c3-48886c07a617" containerName="revoke-keys" Nov 03 00:33:54.535967 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:54.535937 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c84aa5c3-904d-4577-99c3-48886c07a617" containerName="revoke-keys" Nov 03 00:33:54.536060 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:54.536048 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="c84aa5c3-904d-4577-99c3-48886c07a617" containerName="revoke-keys" Nov 03 00:33:54.539466 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:54.539447 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v2ck7" Nov 03 00:33:54.633548 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:54.614621 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v2ck7"] Nov 03 00:33:54.647305 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:54.636867 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4bd4\" (UniqueName: \"kubernetes.io/projected/42293b14-7074-4a8e-90f2-75ce0828dbc8-kube-api-access-p4bd4\") pod \"maas-api-revoke-sub-79805ca9-v2ck7\" (UID: \"42293b14-7074-4a8e-90f2-75ce0828dbc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v2ck7" Nov 03 00:33:54.647674 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:54.647648 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42293b14-7074-4a8e-90f2-75ce0828dbc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-v2ck7\" (UID: \"42293b14-7074-4a8e-90f2-75ce0828dbc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v2ck7" Nov 03 00:33:54.647837 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:54.647821 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42293b14-7074-4a8e-90f2-75ce0828dbc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-v2ck7\" (UID: \"42293b14-7074-4a8e-90f2-75ce0828dbc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v2ck7" Nov 03 00:33:54.672270 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:54.667866 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:33:54.672270 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:54.667937 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:33:54.672270 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:54.668044 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:33:54.672270 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:54.668166 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:33:54.749156 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:54.749094 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42293b14-7074-4a8e-90f2-75ce0828dbc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-v2ck7\" (UID: \"42293b14-7074-4a8e-90f2-75ce0828dbc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v2ck7" Nov 03 00:33:54.754583 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:54.754551 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4bd4\" (UniqueName: \"kubernetes.io/projected/42293b14-7074-4a8e-90f2-75ce0828dbc8-kube-api-access-p4bd4\") pod \"maas-api-revoke-sub-79805ca9-v2ck7\" (UID: \"42293b14-7074-4a8e-90f2-75ce0828dbc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v2ck7" Nov 03 00:33:54.760271 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:54.760244 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42293b14-7074-4a8e-90f2-75ce0828dbc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-v2ck7\" (UID: \"42293b14-7074-4a8e-90f2-75ce0828dbc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v2ck7" Nov 03 00:33:54.770425 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:54.770397 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42293b14-7074-4a8e-90f2-75ce0828dbc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-v2ck7\" (UID: \"42293b14-7074-4a8e-90f2-75ce0828dbc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v2ck7" Nov 03 00:33:54.796442 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:54.790540 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42293b14-7074-4a8e-90f2-75ce0828dbc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-v2ck7\" (UID: \"42293b14-7074-4a8e-90f2-75ce0828dbc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v2ck7" Nov 03 00:33:54.817153 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:54.811850 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4bd4\" (UniqueName: \"kubernetes.io/projected/42293b14-7074-4a8e-90f2-75ce0828dbc8-kube-api-access-p4bd4\") pod \"maas-api-revoke-sub-79805ca9-v2ck7\" (UID: \"42293b14-7074-4a8e-90f2-75ce0828dbc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v2ck7" Nov 03 00:33:54.871427 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:54.871383 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v2ck7" Nov 03 00:33:55.273504 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:55.273471 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v2ck7"] Nov 03 00:33:55.275288 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:33:55.275260 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42293b14_7074_4a8e_90f2_75ce0828dbc8.slice/crio-ad870e0b71f2e3f1f569aa49c27c92f9c782c2059627f853751a5472e38bafed WatchSource:0}: Error finding container ad870e0b71f2e3f1f569aa49c27c92f9c782c2059627f853751a5472e38bafed: Status 404 returned error can't find the container with id ad870e0b71f2e3f1f569aa49c27c92f9c782c2059627f853751a5472e38bafed Nov 03 00:33:55.343350 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:55.343319 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v2ck7" event={"ID":"42293b14-7074-4a8e-90f2-75ce0828dbc8","Type":"ContainerStarted","Data":"ad870e0b71f2e3f1f569aa49c27c92f9c782c2059627f853751a5472e38bafed"} Nov 03 00:33:56.351185 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:56.351121 5727 generic.go:356] "Generic (PLEG): container finished" podID="42293b14-7074-4a8e-90f2-75ce0828dbc8" containerID="4f39ec27302f6c5604ec222aea3610340b3de924dcbededbb64cd92da868060b" exitCode=0 Nov 03 00:33:56.351616 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:56.351202 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v2ck7" event={"ID":"42293b14-7074-4a8e-90f2-75ce0828dbc8","Type":"ContainerDied","Data":"4f39ec27302f6c5604ec222aea3610340b3de924dcbededbb64cd92da868060b"} Nov 03 00:33:57.524091 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:57.524068 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v2ck7" Nov 03 00:33:57.597547 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:57.597516 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42293b14-7074-4a8e-90f2-75ce0828dbc8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42293b14-7074-4a8e-90f2-75ce0828dbc8-maas-api-openshift-ca\") pod \"42293b14-7074-4a8e-90f2-75ce0828dbc8\" (UID: \"42293b14-7074-4a8e-90f2-75ce0828dbc8\") " Nov 03 00:33:57.597715 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:57.597635 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42293b14-7074-4a8e-90f2-75ce0828dbc8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42293b14-7074-4a8e-90f2-75ce0828dbc8-maas-api-xks-ca\") pod \"42293b14-7074-4a8e-90f2-75ce0828dbc8\" (UID: \"42293b14-7074-4a8e-90f2-75ce0828dbc8\") " Nov 03 00:33:57.597760 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:57.597742 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42293b14-7074-4a8e-90f2-75ce0828dbc8-kube-api-access-p4bd4\" (UniqueName: \"kubernetes.io/projected/42293b14-7074-4a8e-90f2-75ce0828dbc8-kube-api-access-p4bd4\") pod \"42293b14-7074-4a8e-90f2-75ce0828dbc8\" (UID: \"42293b14-7074-4a8e-90f2-75ce0828dbc8\") " Nov 03 00:33:57.597947 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:57.597925 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42293b14-7074-4a8e-90f2-75ce0828dbc8-maas-api-openshift-ca" pod "42293b14-7074-4a8e-90f2-75ce0828dbc8" (UID: "42293b14-7074-4a8e-90f2-75ce0828dbc8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:33:57.598102 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:57.598082 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42293b14-7074-4a8e-90f2-75ce0828dbc8-maas-api-openshift-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:57.599857 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:57.599835 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42293b14-7074-4a8e-90f2-75ce0828dbc8-kube-api-access-p4bd4" pod "42293b14-7074-4a8e-90f2-75ce0828dbc8" (UID: "42293b14-7074-4a8e-90f2-75ce0828dbc8"). InnerVolumeSpecName "kube-api-access-p4bd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:33:57.599961 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:57.599945 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42293b14-7074-4a8e-90f2-75ce0828dbc8-maas-api-xks-ca" pod "42293b14-7074-4a8e-90f2-75ce0828dbc8" (UID: "42293b14-7074-4a8e-90f2-75ce0828dbc8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:33:57.699428 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:57.699347 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42293b14-7074-4a8e-90f2-75ce0828dbc8-maas-api-xks-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:57.699428 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:57.699377 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4bd4\" (UniqueName: \"kubernetes.io/projected/42293b14-7074-4a8e-90f2-75ce0828dbc8-kube-api-access-p4bd4\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:57.845191 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:33:57.845122 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:33:58.367620 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:58.367582 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v2ck7" Nov 03 00:33:58.367620 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:58.367605 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v2ck7" event={"ID":"42293b14-7074-4a8e-90f2-75ce0828dbc8","Type":"ContainerDied","Data":"ad870e0b71f2e3f1f569aa49c27c92f9c782c2059627f853751a5472e38bafed"} Nov 03 00:33:58.367836 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:33:58.367656 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad870e0b71f2e3f1f569aa49c27c92f9c782c2059627f853751a5472e38bafed" Nov 03 00:34:11.972638 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:11.972590 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ppzb6"] Nov 03 00:34:11.973558 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:11.973533 5727 cpu_manager.go:414] "containerMap: removing container" podUID="42293b14-7074-4a8e-90f2-75ce0828dbc8" containerName="revoke-keys" Nov 03 00:34:11.973731 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:11.973561 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42293b14-7074-4a8e-90f2-75ce0828dbc8" containerName="revoke-keys" Nov 03 00:34:11.973731 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:11.973701 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="42293b14-7074-4a8e-90f2-75ce0828dbc8" containerName="revoke-keys" Nov 03 00:34:11.977806 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:11.977785 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ppzb6" Nov 03 00:34:12.035893 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:12.035856 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ppzb6"] Nov 03 00:34:12.043599 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:12.043565 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba3816a0-ae12-45ee-848e-c9c2d2ef994d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ppzb6\" (UID: \"ba3816a0-ae12-45ee-848e-c9c2d2ef994d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ppzb6" Nov 03 00:34:12.043856 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:12.043836 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba3816a0-ae12-45ee-848e-c9c2d2ef994d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ppzb6\" (UID: \"ba3816a0-ae12-45ee-848e-c9c2d2ef994d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ppzb6" Nov 03 00:34:12.043996 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:12.043977 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sjrh\" (UniqueName: \"kubernetes.io/projected/ba3816a0-ae12-45ee-848e-c9c2d2ef994d-kube-api-access-7sjrh\") pod \"maas-api-revoke-sub-0b7e8edb-ppzb6\" (UID: \"ba3816a0-ae12-45ee-848e-c9c2d2ef994d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ppzb6" Nov 03 00:34:12.078969 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:12.078939 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:34:12.079168 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:12.078998 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:34:12.079168 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:12.079023 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:34:12.083365 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:12.083346 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:34:12.145783 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:12.145750 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba3816a0-ae12-45ee-848e-c9c2d2ef994d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ppzb6\" (UID: \"ba3816a0-ae12-45ee-848e-c9c2d2ef994d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ppzb6" Nov 03 00:34:12.145923 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:12.145809 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba3816a0-ae12-45ee-848e-c9c2d2ef994d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ppzb6\" (UID: \"ba3816a0-ae12-45ee-848e-c9c2d2ef994d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ppzb6" Nov 03 00:34:12.145923 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:12.145841 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sjrh\" (UniqueName: \"kubernetes.io/projected/ba3816a0-ae12-45ee-848e-c9c2d2ef994d-kube-api-access-7sjrh\") pod \"maas-api-revoke-sub-0b7e8edb-ppzb6\" (UID: \"ba3816a0-ae12-45ee-848e-c9c2d2ef994d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ppzb6" Nov 03 00:34:12.146688 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:12.146614 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba3816a0-ae12-45ee-848e-c9c2d2ef994d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ppzb6\" (UID: \"ba3816a0-ae12-45ee-848e-c9c2d2ef994d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ppzb6" Nov 03 00:34:12.148506 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:12.148481 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba3816a0-ae12-45ee-848e-c9c2d2ef994d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ppzb6\" (UID: \"ba3816a0-ae12-45ee-848e-c9c2d2ef994d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ppzb6" Nov 03 00:34:12.206655 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:12.206625 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sjrh\" (UniqueName: \"kubernetes.io/projected/ba3816a0-ae12-45ee-848e-c9c2d2ef994d-kube-api-access-7sjrh\") pod \"maas-api-revoke-sub-0b7e8edb-ppzb6\" (UID: \"ba3816a0-ae12-45ee-848e-c9c2d2ef994d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ppzb6" Nov 03 00:34:12.299395 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:12.299298 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ppzb6" Nov 03 00:34:12.834078 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:12.834044 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ppzb6"] Nov 03 00:34:13.535437 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:13.535396 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ppzb6" event={"ID":"ba3816a0-ae12-45ee-848e-c9c2d2ef994d","Type":"ContainerStarted","Data":"309066e07edd75733330166a135befe51c9a879b1007dce5cee5e2662aed713b"} Nov 03 00:34:13.657155 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:13.652107 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ppzb6" podStartSLOduration=2.652086341 podStartE2EDuration="2.652086341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:34:13.649879907 +0000 UTC m=+6339.888377597" watchObservedRunningTime="2026-11-03 00:34:13.652086341 +0000 UTC m=+6339.890584005" Nov 03 00:34:14.542805 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:14.542721 5727 generic.go:356] "Generic (PLEG): container finished" podID="ba3816a0-ae12-45ee-848e-c9c2d2ef994d" containerID="bfec6122040e4db0c163d5f8963fa00c14188d4a172aeab5411984c14937512b" exitCode=0 Nov 03 00:34:14.542805 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:14.542766 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ppzb6" event={"ID":"ba3816a0-ae12-45ee-848e-c9c2d2ef994d","Type":"ContainerDied","Data":"bfec6122040e4db0c163d5f8963fa00c14188d4a172aeab5411984c14937512b"} Nov 03 00:34:15.700872 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:15.700831 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ppzb6" Nov 03 00:34:15.767542 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:15.767515 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba3816a0-ae12-45ee-848e-c9c2d2ef994d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba3816a0-ae12-45ee-848e-c9c2d2ef994d-maas-api-openshift-ca\") pod \"ba3816a0-ae12-45ee-848e-c9c2d2ef994d\" (UID: \"ba3816a0-ae12-45ee-848e-c9c2d2ef994d\") " Nov 03 00:34:15.767667 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:15.767560 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba3816a0-ae12-45ee-848e-c9c2d2ef994d-kube-api-access-7sjrh\" (UniqueName: \"kubernetes.io/projected/ba3816a0-ae12-45ee-848e-c9c2d2ef994d-kube-api-access-7sjrh\") pod \"ba3816a0-ae12-45ee-848e-c9c2d2ef994d\" (UID: \"ba3816a0-ae12-45ee-848e-c9c2d2ef994d\") " Nov 03 00:34:15.767719 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:15.767684 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba3816a0-ae12-45ee-848e-c9c2d2ef994d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba3816a0-ae12-45ee-848e-c9c2d2ef994d-maas-api-xks-ca\") pod \"ba3816a0-ae12-45ee-848e-c9c2d2ef994d\" (UID: \"ba3816a0-ae12-45ee-848e-c9c2d2ef994d\") " Nov 03 00:34:15.767890 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:15.767863 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba3816a0-ae12-45ee-848e-c9c2d2ef994d-maas-api-openshift-ca" pod "ba3816a0-ae12-45ee-848e-c9c2d2ef994d" (UID: "ba3816a0-ae12-45ee-848e-c9c2d2ef994d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:34:15.768054 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:15.768036 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba3816a0-ae12-45ee-848e-c9c2d2ef994d-maas-api-openshift-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:15.769875 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:15.769851 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba3816a0-ae12-45ee-848e-c9c2d2ef994d-maas-api-xks-ca" pod "ba3816a0-ae12-45ee-848e-c9c2d2ef994d" (UID: "ba3816a0-ae12-45ee-848e-c9c2d2ef994d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:34:15.769987 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:15.769875 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3816a0-ae12-45ee-848e-c9c2d2ef994d-kube-api-access-7sjrh" pod "ba3816a0-ae12-45ee-848e-c9c2d2ef994d" (UID: "ba3816a0-ae12-45ee-848e-c9c2d2ef994d"). InnerVolumeSpecName "kube-api-access-7sjrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:34:15.868868 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:15.868835 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba3816a0-ae12-45ee-848e-c9c2d2ef994d-maas-api-xks-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:15.868868 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:15.868865 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sjrh\" (UniqueName: \"kubernetes.io/projected/ba3816a0-ae12-45ee-848e-c9c2d2ef994d-kube-api-access-7sjrh\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:16.557374 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:16.557341 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ppzb6" Nov 03 00:34:16.557559 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:16.557375 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ppzb6" event={"ID":"ba3816a0-ae12-45ee-848e-c9c2d2ef994d","Type":"ContainerDied","Data":"309066e07edd75733330166a135befe51c9a879b1007dce5cee5e2662aed713b"} Nov 03 00:34:16.557559 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:16.557429 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="309066e07edd75733330166a135befe51c9a879b1007dce5cee5e2662aed713b" Nov 03 00:34:23.270490 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:23.270458 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29rtv"] Nov 03 00:34:23.270989 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:23.270927 5727 cpu_manager.go:414] "containerMap: removing container" podUID="ba3816a0-ae12-45ee-848e-c9c2d2ef994d" containerName="revoke-keys" Nov 03 00:34:23.270989 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:23.270938 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba3816a0-ae12-45ee-848e-c9c2d2ef994d" containerName="revoke-keys" Nov 03 00:34:23.271101 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:23.271027 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba3816a0-ae12-45ee-848e-c9c2d2ef994d" containerName="revoke-keys" Nov 03 00:34:23.274453 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:23.274433 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29rtv" Nov 03 00:34:23.327893 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:23.327862 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:34:23.343425 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:23.343394 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm4pr\" (UniqueName: \"kubernetes.io/projected/831d6d01-6cee-440a-b927-af1b347e4931-kube-api-access-vm4pr\") pod \"maas-api-revoke-sub-3d4156f1-29rtv\" (UID: \"831d6d01-6cee-440a-b927-af1b347e4931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29rtv" Nov 03 00:34:23.343599 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:23.343573 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/831d6d01-6cee-440a-b927-af1b347e4931-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-29rtv\" (UID: \"831d6d01-6cee-440a-b927-af1b347e4931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29rtv" Nov 03 00:34:23.343659 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:23.343618 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/831d6d01-6cee-440a-b927-af1b347e4931-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-29rtv\" (UID: \"831d6d01-6cee-440a-b927-af1b347e4931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29rtv" Nov 03 00:34:23.377320 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:23.377292 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:34:23.377474 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:23.377338 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:34:23.377474 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:23.377412 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:34:23.380494 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:23.380471 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29rtv"] Nov 03 00:34:23.444715 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:23.444683 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm4pr\" (UniqueName: \"kubernetes.io/projected/831d6d01-6cee-440a-b927-af1b347e4931-kube-api-access-vm4pr\") pod \"maas-api-revoke-sub-3d4156f1-29rtv\" (UID: \"831d6d01-6cee-440a-b927-af1b347e4931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29rtv" Nov 03 00:34:23.444836 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:23.444790 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/831d6d01-6cee-440a-b927-af1b347e4931-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-29rtv\" (UID: \"831d6d01-6cee-440a-b927-af1b347e4931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29rtv" Nov 03 00:34:23.444836 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:23.444820 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/831d6d01-6cee-440a-b927-af1b347e4931-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-29rtv\" (UID: \"831d6d01-6cee-440a-b927-af1b347e4931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29rtv" Nov 03 00:34:23.445468 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:23.445443 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/831d6d01-6cee-440a-b927-af1b347e4931-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-29rtv\" (UID: \"831d6d01-6cee-440a-b927-af1b347e4931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29rtv" Nov 03 00:34:23.447360 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:23.447338 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/831d6d01-6cee-440a-b927-af1b347e4931-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-29rtv\" (UID: \"831d6d01-6cee-440a-b927-af1b347e4931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29rtv" Nov 03 00:34:23.505532 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:23.505510 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm4pr\" (UniqueName: \"kubernetes.io/projected/831d6d01-6cee-440a-b927-af1b347e4931-kube-api-access-vm4pr\") pod \"maas-api-revoke-sub-3d4156f1-29rtv\" (UID: \"831d6d01-6cee-440a-b927-af1b347e4931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29rtv" Nov 03 00:34:23.588574 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:23.588546 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29rtv" Nov 03 00:34:23.968200 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:23.968162 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29rtv"] Nov 03 00:34:23.969989 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:34:23.969955 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod831d6d01_6cee_440a_b927_af1b347e4931.slice/crio-d359c8b97959472ee8c95cc14022e85cba02f4f5b48d41b8a84b7847b2f6c012 WatchSource:0}: Error finding container d359c8b97959472ee8c95cc14022e85cba02f4f5b48d41b8a84b7847b2f6c012: Status 404 returned error can't find the container with id d359c8b97959472ee8c95cc14022e85cba02f4f5b48d41b8a84b7847b2f6c012 Nov 03 00:34:24.615896 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:24.615853 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29rtv" event={"ID":"831d6d01-6cee-440a-b927-af1b347e4931","Type":"ContainerStarted","Data":"fbe7dab0411f2ad44ef2718f3993574b0ae513d22a29a11918ce5b320d5155da"} Nov 03 00:34:24.616410 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:24.615914 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29rtv" event={"ID":"831d6d01-6cee-440a-b927-af1b347e4931","Type":"ContainerStarted","Data":"d359c8b97959472ee8c95cc14022e85cba02f4f5b48d41b8a84b7847b2f6c012"} Nov 03 00:34:24.733815 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:24.733764 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29rtv" podStartSLOduration=1.733749166 podStartE2EDuration="1.733749166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:34:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:34:24.73059317 +0000 UTC m=+6350.969090849" watchObservedRunningTime="2026-11-03 00:34:24.733749166 +0000 UTC m=+6350.972246825" Nov 03 00:34:25.624349 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:25.624306 5727 generic.go:356] "Generic (PLEG): container finished" podID="831d6d01-6cee-440a-b927-af1b347e4931" containerID="fbe7dab0411f2ad44ef2718f3993574b0ae513d22a29a11918ce5b320d5155da" exitCode=0 Nov 03 00:34:25.624754 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:25.624392 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29rtv" event={"ID":"831d6d01-6cee-440a-b927-af1b347e4931","Type":"ContainerDied","Data":"fbe7dab0411f2ad44ef2718f3993574b0ae513d22a29a11918ce5b320d5155da"} Nov 03 00:34:26.785194 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:26.785165 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29rtv" Nov 03 00:34:26.880760 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:26.880732 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/831d6d01-6cee-440a-b927-af1b347e4931-kube-api-access-vm4pr\" (UniqueName: \"kubernetes.io/projected/831d6d01-6cee-440a-b927-af1b347e4931-kube-api-access-vm4pr\") pod \"831d6d01-6cee-440a-b927-af1b347e4931\" (UID: \"831d6d01-6cee-440a-b927-af1b347e4931\") " Nov 03 00:34:26.880902 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:26.880820 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/831d6d01-6cee-440a-b927-af1b347e4931-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/831d6d01-6cee-440a-b927-af1b347e4931-maas-api-openshift-ca\") pod \"831d6d01-6cee-440a-b927-af1b347e4931\" (UID: \"831d6d01-6cee-440a-b927-af1b347e4931\") " Nov 03 00:34:26.880902 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:26.880845 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/831d6d01-6cee-440a-b927-af1b347e4931-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/831d6d01-6cee-440a-b927-af1b347e4931-maas-api-xks-ca\") pod \"831d6d01-6cee-440a-b927-af1b347e4931\" (UID: \"831d6d01-6cee-440a-b927-af1b347e4931\") " Nov 03 00:34:26.881188 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:26.881108 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/831d6d01-6cee-440a-b927-af1b347e4931-maas-api-openshift-ca" pod "831d6d01-6cee-440a-b927-af1b347e4931" (UID: "831d6d01-6cee-440a-b927-af1b347e4931"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:34:26.883052 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:26.883026 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/831d6d01-6cee-440a-b927-af1b347e4931-kube-api-access-vm4pr" pod "831d6d01-6cee-440a-b927-af1b347e4931" (UID: "831d6d01-6cee-440a-b927-af1b347e4931"). InnerVolumeSpecName "kube-api-access-vm4pr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:34:26.883052 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:26.883036 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/831d6d01-6cee-440a-b927-af1b347e4931-maas-api-xks-ca" pod "831d6d01-6cee-440a-b927-af1b347e4931" (UID: "831d6d01-6cee-440a-b927-af1b347e4931"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:34:26.982210 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:26.982071 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/831d6d01-6cee-440a-b927-af1b347e4931-maas-api-openshift-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:26.982210 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:26.982106 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/831d6d01-6cee-440a-b927-af1b347e4931-maas-api-xks-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:26.982210 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:26.982123 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm4pr\" (UniqueName: \"kubernetes.io/projected/831d6d01-6cee-440a-b927-af1b347e4931-kube-api-access-vm4pr\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:27.638870 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:27.638843 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29rtv" Nov 03 00:34:27.639053 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:27.638843 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29rtv" event={"ID":"831d6d01-6cee-440a-b927-af1b347e4931","Type":"ContainerDied","Data":"d359c8b97959472ee8c95cc14022e85cba02f4f5b48d41b8a84b7847b2f6c012"} Nov 03 00:34:27.639053 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:27.638962 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d359c8b97959472ee8c95cc14022e85cba02f4f5b48d41b8a84b7847b2f6c012" Nov 03 00:34:27.845276 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:34:27.845233 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:34:36.417902 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:36.417869 5727 scope.go:122] "RemoveContainer" containerID="db71d459959172e6d86b517a15eade9e533ee65958a7a42cf8a92475ebd14596" Nov 03 00:34:47.581846 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:47.581812 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29rtv"] Nov 03 00:34:47.631328 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:47.631290 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29rtv"] Nov 03 00:34:47.704477 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:47.704444 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzkqq"] Nov 03 00:34:47.705177 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:47.705157 5727 cpu_manager.go:414] "containerMap: removing container" podUID="831d6d01-6cee-440a-b927-af1b347e4931" containerName="revoke-keys" Nov 03 00:34:47.705285 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:47.705178 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="831d6d01-6cee-440a-b927-af1b347e4931" containerName="revoke-keys" Nov 03 00:34:47.705343 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:47.705334 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="831d6d01-6cee-440a-b927-af1b347e4931" containerName="revoke-keys" Nov 03 00:34:47.710488 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:47.710466 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzkqq" Nov 03 00:34:47.737983 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:47.737957 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/461b0b8f-3f8b-4efa-9de5-340f33232caf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wzkqq\" (UID: \"461b0b8f-3f8b-4efa-9de5-340f33232caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzkqq" Nov 03 00:34:47.738087 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:47.738002 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkkxf\" (UniqueName: \"kubernetes.io/projected/461b0b8f-3f8b-4efa-9de5-340f33232caf-kube-api-access-gkkxf\") pod \"maas-api-revoke-sub-3d4156f1-wzkqq\" (UID: \"461b0b8f-3f8b-4efa-9de5-340f33232caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzkqq" Nov 03 00:34:47.738170 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:47.738119 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/461b0b8f-3f8b-4efa-9de5-340f33232caf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wzkqq\" (UID: \"461b0b8f-3f8b-4efa-9de5-340f33232caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzkqq" Nov 03 00:34:47.762789 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:47.762605 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:34:47.762789 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:47.762713 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:34:47.762789 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:47.762747 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:34:47.763042 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:47.762861 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:34:47.763499 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:47.763479 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzkqq"] Nov 03 00:34:47.838920 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:47.838833 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/461b0b8f-3f8b-4efa-9de5-340f33232caf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wzkqq\" (UID: \"461b0b8f-3f8b-4efa-9de5-340f33232caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzkqq" Nov 03 00:34:47.838920 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:47.838880 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkkxf\" (UniqueName: \"kubernetes.io/projected/461b0b8f-3f8b-4efa-9de5-340f33232caf-kube-api-access-gkkxf\") pod \"maas-api-revoke-sub-3d4156f1-wzkqq\" (UID: \"461b0b8f-3f8b-4efa-9de5-340f33232caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzkqq" Nov 03 00:34:47.838920 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:47.838913 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/461b0b8f-3f8b-4efa-9de5-340f33232caf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wzkqq\" (UID: \"461b0b8f-3f8b-4efa-9de5-340f33232caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzkqq" Nov 03 00:34:47.839588 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:47.839568 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/461b0b8f-3f8b-4efa-9de5-340f33232caf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wzkqq\" (UID: \"461b0b8f-3f8b-4efa-9de5-340f33232caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzkqq" Nov 03 00:34:47.841413 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:47.841390 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/461b0b8f-3f8b-4efa-9de5-340f33232caf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wzkqq\" (UID: \"461b0b8f-3f8b-4efa-9de5-340f33232caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzkqq" Nov 03 00:34:47.882431 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:47.882397 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="831d6d01-6cee-440a-b927-af1b347e4931" path="/var/lib/kubelet/pods/831d6d01-6cee-440a-b927-af1b347e4931/volumes" Nov 03 00:34:47.899798 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:47.899770 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkkxf\" (UniqueName: \"kubernetes.io/projected/461b0b8f-3f8b-4efa-9de5-340f33232caf-kube-api-access-gkkxf\") pod \"maas-api-revoke-sub-3d4156f1-wzkqq\" (UID: \"461b0b8f-3f8b-4efa-9de5-340f33232caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzkqq" Nov 03 00:34:48.023727 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:48.023699 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzkqq" Nov 03 00:34:48.400038 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:48.400011 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzkqq"] Nov 03 00:34:48.401820 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:34:48.401794 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod461b0b8f_3f8b_4efa_9de5_340f33232caf.slice/crio-ae159b410db00aa3871fa86ca7b305f4252329b39577674107ab1cae70d32552 WatchSource:0}: Error finding container ae159b410db00aa3871fa86ca7b305f4252329b39577674107ab1cae70d32552: Status 404 returned error can't find the container with id ae159b410db00aa3871fa86ca7b305f4252329b39577674107ab1cae70d32552 Nov 03 00:34:48.824855 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:48.824818 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzkqq" event={"ID":"461b0b8f-3f8b-4efa-9de5-340f33232caf","Type":"ContainerStarted","Data":"39cde42db47ddc51dce3739e7e0bcacd361e241fc89ac5d5e2b3fc0e7478e0fd"} Nov 03 00:34:48.825328 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:48.824872 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzkqq" event={"ID":"461b0b8f-3f8b-4efa-9de5-340f33232caf","Type":"ContainerStarted","Data":"ae159b410db00aa3871fa86ca7b305f4252329b39577674107ab1cae70d32552"} Nov 03 00:34:48.940548 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:48.940451 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzkqq" podStartSLOduration=1.940435039 podStartE2EDuration="1.940435039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:34:48.93859619 +0000 UTC m=+6375.177093853" watchObservedRunningTime="2026-11-03 00:34:48.940435039 +0000 UTC m=+6375.178932701" Nov 03 00:34:49.833184 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:49.833147 5727 generic.go:356] "Generic (PLEG): container finished" podID="461b0b8f-3f8b-4efa-9de5-340f33232caf" containerID="39cde42db47ddc51dce3739e7e0bcacd361e241fc89ac5d5e2b3fc0e7478e0fd" exitCode=0 Nov 03 00:34:49.833637 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:49.833240 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzkqq" event={"ID":"461b0b8f-3f8b-4efa-9de5-340f33232caf","Type":"ContainerDied","Data":"39cde42db47ddc51dce3739e7e0bcacd361e241fc89ac5d5e2b3fc0e7478e0fd"} Nov 03 00:34:50.993595 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:50.993569 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzkqq" Nov 03 00:34:51.069278 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:51.069246 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/461b0b8f-3f8b-4efa-9de5-340f33232caf-kube-api-access-gkkxf\" (UniqueName: \"kubernetes.io/projected/461b0b8f-3f8b-4efa-9de5-340f33232caf-kube-api-access-gkkxf\") pod \"461b0b8f-3f8b-4efa-9de5-340f33232caf\" (UID: \"461b0b8f-3f8b-4efa-9de5-340f33232caf\") " Nov 03 00:34:51.069456 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:51.069438 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/461b0b8f-3f8b-4efa-9de5-340f33232caf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/461b0b8f-3f8b-4efa-9de5-340f33232caf-maas-api-xks-ca\") pod \"461b0b8f-3f8b-4efa-9de5-340f33232caf\" (UID: \"461b0b8f-3f8b-4efa-9de5-340f33232caf\") " Nov 03 00:34:51.069538 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:51.069486 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/461b0b8f-3f8b-4efa-9de5-340f33232caf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/461b0b8f-3f8b-4efa-9de5-340f33232caf-maas-api-openshift-ca\") pod \"461b0b8f-3f8b-4efa-9de5-340f33232caf\" (UID: \"461b0b8f-3f8b-4efa-9de5-340f33232caf\") " Nov 03 00:34:51.069838 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:51.069817 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/461b0b8f-3f8b-4efa-9de5-340f33232caf-maas-api-openshift-ca" pod "461b0b8f-3f8b-4efa-9de5-340f33232caf" (UID: "461b0b8f-3f8b-4efa-9de5-340f33232caf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:34:51.071528 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:51.071500 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/461b0b8f-3f8b-4efa-9de5-340f33232caf-kube-api-access-gkkxf" pod "461b0b8f-3f8b-4efa-9de5-340f33232caf" (UID: "461b0b8f-3f8b-4efa-9de5-340f33232caf"). InnerVolumeSpecName "kube-api-access-gkkxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:34:51.071671 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:51.071647 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/461b0b8f-3f8b-4efa-9de5-340f33232caf-maas-api-xks-ca" pod "461b0b8f-3f8b-4efa-9de5-340f33232caf" (UID: "461b0b8f-3f8b-4efa-9de5-340f33232caf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:34:51.170468 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:51.170384 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkkxf\" (UniqueName: \"kubernetes.io/projected/461b0b8f-3f8b-4efa-9de5-340f33232caf-kube-api-access-gkkxf\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:51.170468 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:51.170416 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/461b0b8f-3f8b-4efa-9de5-340f33232caf-maas-api-xks-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:51.170468 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:51.170440 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/461b0b8f-3f8b-4efa-9de5-340f33232caf-maas-api-openshift-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:51.852748 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:51.852712 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzkqq" event={"ID":"461b0b8f-3f8b-4efa-9de5-340f33232caf","Type":"ContainerDied","Data":"ae159b410db00aa3871fa86ca7b305f4252329b39577674107ab1cae70d32552"} Nov 03 00:34:51.852748 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:51.852736 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzkqq" Nov 03 00:34:51.853025 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:34:51.852792 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae159b410db00aa3871fa86ca7b305f4252329b39577674107ab1cae70d32552" Nov 03 00:34:57.845435 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:34:57.845399 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:35:27.845147 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:35:27.845090 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:35:33.115625 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:33.115587 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6zk75"] Nov 03 00:35:33.116367 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:33.116347 5727 cpu_manager.go:414] "containerMap: removing container" podUID="461b0b8f-3f8b-4efa-9de5-340f33232caf" containerName="revoke-keys" Nov 03 00:35:33.116447 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:33.116371 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="461b0b8f-3f8b-4efa-9de5-340f33232caf" containerName="revoke-keys" Nov 03 00:35:33.116507 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:33.116494 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="461b0b8f-3f8b-4efa-9de5-340f33232caf" containerName="revoke-keys" Nov 03 00:35:33.120028 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:33.120007 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6zk75" Nov 03 00:35:33.177418 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:33.177382 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6zk75"] Nov 03 00:35:33.221455 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:33.221422 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:35:33.221637 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:33.221442 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:35:33.221637 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:33.221482 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:35:33.221637 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:33.221502 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:35:33.251877 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:33.251835 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1828ef7-b027-42f5-b705-6e37ff37b54c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6zk75\" (UID: \"a1828ef7-b027-42f5-b705-6e37ff37b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6zk75" Nov 03 00:35:33.252041 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:33.251971 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1828ef7-b027-42f5-b705-6e37ff37b54c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6zk75\" (UID: \"a1828ef7-b027-42f5-b705-6e37ff37b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6zk75" Nov 03 00:35:33.252185 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:33.252159 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2dsz\" (UniqueName: \"kubernetes.io/projected/a1828ef7-b027-42f5-b705-6e37ff37b54c-kube-api-access-n2dsz\") pod \"maas-api-revoke-sub-e2fc9c5b-6zk75\" (UID: \"a1828ef7-b027-42f5-b705-6e37ff37b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6zk75" Nov 03 00:35:33.353672 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:33.353635 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1828ef7-b027-42f5-b705-6e37ff37b54c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6zk75\" (UID: \"a1828ef7-b027-42f5-b705-6e37ff37b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6zk75" Nov 03 00:35:33.353852 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:33.353726 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2dsz\" (UniqueName: \"kubernetes.io/projected/a1828ef7-b027-42f5-b705-6e37ff37b54c-kube-api-access-n2dsz\") pod \"maas-api-revoke-sub-e2fc9c5b-6zk75\" (UID: \"a1828ef7-b027-42f5-b705-6e37ff37b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6zk75" Nov 03 00:35:33.353852 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:33.353771 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1828ef7-b027-42f5-b705-6e37ff37b54c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6zk75\" (UID: \"a1828ef7-b027-42f5-b705-6e37ff37b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6zk75" Nov 03 00:35:33.354475 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:33.354445 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1828ef7-b027-42f5-b705-6e37ff37b54c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6zk75\" (UID: \"a1828ef7-b027-42f5-b705-6e37ff37b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6zk75" Nov 03 00:35:33.356365 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:33.356340 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1828ef7-b027-42f5-b705-6e37ff37b54c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6zk75\" (UID: \"a1828ef7-b027-42f5-b705-6e37ff37b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6zk75" Nov 03 00:35:33.414606 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:33.414539 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2dsz\" (UniqueName: \"kubernetes.io/projected/a1828ef7-b027-42f5-b705-6e37ff37b54c-kube-api-access-n2dsz\") pod \"maas-api-revoke-sub-e2fc9c5b-6zk75\" (UID: \"a1828ef7-b027-42f5-b705-6e37ff37b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6zk75" Nov 03 00:35:33.433479 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:33.433456 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6zk75" Nov 03 00:35:33.816297 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:33.816267 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6zk75"] Nov 03 00:35:33.817713 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:35:33.817687 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1828ef7_b027_42f5_b705_6e37ff37b54c.slice/crio-6b4350dd75830d31827cd678552352793afd6403d28607047b851acdc7782325 WatchSource:0}: Error finding container 6b4350dd75830d31827cd678552352793afd6403d28607047b851acdc7782325: Status 404 returned error can't find the container with id 6b4350dd75830d31827cd678552352793afd6403d28607047b851acdc7782325 Nov 03 00:35:34.138774 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:34.138735 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6zk75" event={"ID":"a1828ef7-b027-42f5-b705-6e37ff37b54c","Type":"ContainerStarted","Data":"6b4350dd75830d31827cd678552352793afd6403d28607047b851acdc7782325"} Nov 03 00:35:35.147381 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:35.147340 5727 generic.go:356] "Generic (PLEG): container finished" podID="a1828ef7-b027-42f5-b705-6e37ff37b54c" containerID="0c6f547db2badbdc0e996c435be14a5adf2047428b8bd800847cebfad1e88e5d" exitCode=0 Nov 03 00:35:35.147811 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:35.147428 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6zk75" event={"ID":"a1828ef7-b027-42f5-b705-6e37ff37b54c","Type":"ContainerDied","Data":"0c6f547db2badbdc0e996c435be14a5adf2047428b8bd800847cebfad1e88e5d"} Nov 03 00:35:36.301155 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:36.301100 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6zk75" Nov 03 00:35:36.383356 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:36.383323 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1828ef7-b027-42f5-b705-6e37ff37b54c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1828ef7-b027-42f5-b705-6e37ff37b54c-maas-api-xks-ca\") pod \"a1828ef7-b027-42f5-b705-6e37ff37b54c\" (UID: \"a1828ef7-b027-42f5-b705-6e37ff37b54c\") " Nov 03 00:35:36.383569 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:36.383475 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1828ef7-b027-42f5-b705-6e37ff37b54c-kube-api-access-n2dsz\" (UniqueName: \"kubernetes.io/projected/a1828ef7-b027-42f5-b705-6e37ff37b54c-kube-api-access-n2dsz\") pod \"a1828ef7-b027-42f5-b705-6e37ff37b54c\" (UID: \"a1828ef7-b027-42f5-b705-6e37ff37b54c\") " Nov 03 00:35:36.383569 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:36.383512 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1828ef7-b027-42f5-b705-6e37ff37b54c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1828ef7-b027-42f5-b705-6e37ff37b54c-maas-api-openshift-ca\") pod \"a1828ef7-b027-42f5-b705-6e37ff37b54c\" (UID: \"a1828ef7-b027-42f5-b705-6e37ff37b54c\") " Nov 03 00:35:36.383953 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:36.383928 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1828ef7-b027-42f5-b705-6e37ff37b54c-maas-api-openshift-ca" pod "a1828ef7-b027-42f5-b705-6e37ff37b54c" (UID: "a1828ef7-b027-42f5-b705-6e37ff37b54c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:35:36.385748 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:36.385719 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1828ef7-b027-42f5-b705-6e37ff37b54c-maas-api-xks-ca" pod "a1828ef7-b027-42f5-b705-6e37ff37b54c" (UID: "a1828ef7-b027-42f5-b705-6e37ff37b54c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:35:36.385844 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:36.385790 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1828ef7-b027-42f5-b705-6e37ff37b54c-kube-api-access-n2dsz" pod "a1828ef7-b027-42f5-b705-6e37ff37b54c" (UID: "a1828ef7-b027-42f5-b705-6e37ff37b54c"). InnerVolumeSpecName "kube-api-access-n2dsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:35:36.484867 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:36.484791 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1828ef7-b027-42f5-b705-6e37ff37b54c-maas-api-openshift-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:36.484867 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:36.484821 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1828ef7-b027-42f5-b705-6e37ff37b54c-maas-api-xks-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:36.484867 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:36.484832 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2dsz\" (UniqueName: \"kubernetes.io/projected/a1828ef7-b027-42f5-b705-6e37ff37b54c-kube-api-access-n2dsz\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:37.162395 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:37.162361 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6zk75" Nov 03 00:35:37.162395 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:37.162377 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6zk75" event={"ID":"a1828ef7-b027-42f5-b705-6e37ff37b54c","Type":"ContainerDied","Data":"6b4350dd75830d31827cd678552352793afd6403d28607047b851acdc7782325"} Nov 03 00:35:37.162624 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:37.162429 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b4350dd75830d31827cd678552352793afd6403d28607047b851acdc7782325" Nov 03 00:35:43.931509 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:43.931471 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xbnh9"] Nov 03 00:35:43.931966 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:43.931953 5727 cpu_manager.go:414] "containerMap: removing container" podUID="a1828ef7-b027-42f5-b705-6e37ff37b54c" containerName="revoke-keys" Nov 03 00:35:43.932011 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:43.931967 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1828ef7-b027-42f5-b705-6e37ff37b54c" containerName="revoke-keys" Nov 03 00:35:43.932058 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:43.932048 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1828ef7-b027-42f5-b705-6e37ff37b54c" containerName="revoke-keys" Nov 03 00:35:43.935418 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:43.935395 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xbnh9" Nov 03 00:35:43.988725 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:43.988689 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:35:43.993311 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:43.993285 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xbnh9"] Nov 03 00:35:44.038269 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:44.038235 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:35:44.038412 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:44.038267 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:35:44.038412 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:44.038312 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:35:44.062025 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:44.061998 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwntf\" (UniqueName: \"kubernetes.io/projected/481ba507-2925-4706-947c-898c75c7c210-kube-api-access-jwntf\") pod \"maas-api-revoke-sub-5a29efb3-xbnh9\" (UID: \"481ba507-2925-4706-947c-898c75c7c210\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xbnh9" Nov 03 00:35:44.062272 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:44.062255 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481ba507-2925-4706-947c-898c75c7c210-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-xbnh9\" (UID: \"481ba507-2925-4706-947c-898c75c7c210\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xbnh9" Nov 03 00:35:44.062327 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:44.062294 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481ba507-2925-4706-947c-898c75c7c210-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-xbnh9\" (UID: \"481ba507-2925-4706-947c-898c75c7c210\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xbnh9" Nov 03 00:35:44.163559 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:44.163530 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481ba507-2925-4706-947c-898c75c7c210-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-xbnh9\" (UID: \"481ba507-2925-4706-947c-898c75c7c210\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xbnh9" Nov 03 00:35:44.163704 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:44.163572 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481ba507-2925-4706-947c-898c75c7c210-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-xbnh9\" (UID: \"481ba507-2925-4706-947c-898c75c7c210\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xbnh9" Nov 03 00:35:44.163704 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:44.163612 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwntf\" (UniqueName: \"kubernetes.io/projected/481ba507-2925-4706-947c-898c75c7c210-kube-api-access-jwntf\") pod \"maas-api-revoke-sub-5a29efb3-xbnh9\" (UID: \"481ba507-2925-4706-947c-898c75c7c210\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xbnh9" Nov 03 00:35:44.164326 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:44.164307 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481ba507-2925-4706-947c-898c75c7c210-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-xbnh9\" (UID: \"481ba507-2925-4706-947c-898c75c7c210\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xbnh9" Nov 03 00:35:44.166148 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:44.166116 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481ba507-2925-4706-947c-898c75c7c210-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-xbnh9\" (UID: \"481ba507-2925-4706-947c-898c75c7c210\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xbnh9" Nov 03 00:35:44.224233 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:44.224161 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwntf\" (UniqueName: \"kubernetes.io/projected/481ba507-2925-4706-947c-898c75c7c210-kube-api-access-jwntf\") pod \"maas-api-revoke-sub-5a29efb3-xbnh9\" (UID: \"481ba507-2925-4706-947c-898c75c7c210\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xbnh9" Nov 03 00:35:44.249247 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:44.249218 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xbnh9" Nov 03 00:35:44.624875 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:44.624847 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xbnh9"] Nov 03 00:35:44.626088 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:35:44.626059 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod481ba507_2925_4706_947c_898c75c7c210.slice/crio-0efa76aeeedb6d4b405d70ab4bd6927458cd12dafae1337ac4eda33afef5b8d6 WatchSource:0}: Error finding container 0efa76aeeedb6d4b405d70ab4bd6927458cd12dafae1337ac4eda33afef5b8d6: Status 404 returned error can't find the container with id 0efa76aeeedb6d4b405d70ab4bd6927458cd12dafae1337ac4eda33afef5b8d6 Nov 03 00:35:45.216186 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:45.216081 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xbnh9" event={"ID":"481ba507-2925-4706-947c-898c75c7c210","Type":"ContainerStarted","Data":"d74db0d33b70357fcb9565c58b8652fc8cd0f8abf6bbafa142b014c62b63902b"} Nov 03 00:35:45.216186 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:45.216174 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xbnh9" event={"ID":"481ba507-2925-4706-947c-898c75c7c210","Type":"ContainerStarted","Data":"0efa76aeeedb6d4b405d70ab4bd6927458cd12dafae1337ac4eda33afef5b8d6"} Nov 03 00:35:45.331888 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:45.331832 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xbnh9" podStartSLOduration=2.3318160199999998 podStartE2EDuration="2.33181602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:45.33054785 +0000 UTC m=+6431.569045513" watchObservedRunningTime="2026-11-03 00:35:45.33181602 +0000 UTC m=+6431.570313681" Nov 03 00:35:46.225838 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:46.225803 5727 generic.go:356] "Generic (PLEG): container finished" podID="481ba507-2925-4706-947c-898c75c7c210" containerID="d74db0d33b70357fcb9565c58b8652fc8cd0f8abf6bbafa142b014c62b63902b" exitCode=0 Nov 03 00:35:46.226272 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:46.225859 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xbnh9" event={"ID":"481ba507-2925-4706-947c-898c75c7c210","Type":"ContainerDied","Data":"d74db0d33b70357fcb9565c58b8652fc8cd0f8abf6bbafa142b014c62b63902b"} Nov 03 00:35:47.743810 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:47.743781 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xbnh9" Nov 03 00:35:47.837276 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:47.837239 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/481ba507-2925-4706-947c-898c75c7c210-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481ba507-2925-4706-947c-898c75c7c210-maas-api-xks-ca\") pod \"481ba507-2925-4706-947c-898c75c7c210\" (UID: \"481ba507-2925-4706-947c-898c75c7c210\") " Nov 03 00:35:47.837510 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:47.837324 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/481ba507-2925-4706-947c-898c75c7c210-kube-api-access-jwntf\" (UniqueName: \"kubernetes.io/projected/481ba507-2925-4706-947c-898c75c7c210-kube-api-access-jwntf\") pod \"481ba507-2925-4706-947c-898c75c7c210\" (UID: \"481ba507-2925-4706-947c-898c75c7c210\") " Nov 03 00:35:47.837510 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:47.837455 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/481ba507-2925-4706-947c-898c75c7c210-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481ba507-2925-4706-947c-898c75c7c210-maas-api-openshift-ca\") pod \"481ba507-2925-4706-947c-898c75c7c210\" (UID: \"481ba507-2925-4706-947c-898c75c7c210\") " Nov 03 00:35:47.838038 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:47.838000 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/481ba507-2925-4706-947c-898c75c7c210-maas-api-openshift-ca" pod "481ba507-2925-4706-947c-898c75c7c210" (UID: "481ba507-2925-4706-947c-898c75c7c210"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:35:47.838189 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:47.838105 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481ba507-2925-4706-947c-898c75c7c210-maas-api-openshift-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:47.840307 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:47.840277 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/481ba507-2925-4706-947c-898c75c7c210-kube-api-access-jwntf" pod "481ba507-2925-4706-947c-898c75c7c210" (UID: "481ba507-2925-4706-947c-898c75c7c210"). InnerVolumeSpecName "kube-api-access-jwntf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:35:47.841836 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:47.841803 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/481ba507-2925-4706-947c-898c75c7c210-maas-api-xks-ca" pod "481ba507-2925-4706-947c-898c75c7c210" (UID: "481ba507-2925-4706-947c-898c75c7c210"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:35:47.946508 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:47.946425 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwntf\" (UniqueName: \"kubernetes.io/projected/481ba507-2925-4706-947c-898c75c7c210-kube-api-access-jwntf\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:47.946749 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:47.946727 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481ba507-2925-4706-947c-898c75c7c210-maas-api-xks-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:48.263207 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:48.261187 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xbnh9" event={"ID":"481ba507-2925-4706-947c-898c75c7c210","Type":"ContainerDied","Data":"0efa76aeeedb6d4b405d70ab4bd6927458cd12dafae1337ac4eda33afef5b8d6"} Nov 03 00:35:48.263207 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:48.261274 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0efa76aeeedb6d4b405d70ab4bd6927458cd12dafae1337ac4eda33afef5b8d6" Nov 03 00:35:48.263207 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:48.261369 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xbnh9" Nov 03 00:35:49.046953 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:49.046914 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-29fsr"] Nov 03 00:35:49.096729 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:49.096699 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-29fsr"] Nov 03 00:35:49.882118 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:35:49.882084 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8a714c1-5619-4456-863a-cf9c0e1a250a" path="/var/lib/kubelet/pods/c8a714c1-5619-4456-863a-cf9c0e1a250a/volumes" Nov 03 00:35:57.846290 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:35:57.846251 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:36:00.527572 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:00.527537 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzkqq"] Nov 03 00:36:00.576555 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:00.576521 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzkqq"] Nov 03 00:36:00.646467 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:00.646431 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2mxgc"] Nov 03 00:36:00.646982 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:00.646963 5727 cpu_manager.go:414] "containerMap: removing container" podUID="481ba507-2925-4706-947c-898c75c7c210" containerName="revoke-keys" Nov 03 00:36:00.647084 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:00.646984 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="481ba507-2925-4706-947c-898c75c7c210" containerName="revoke-keys" Nov 03 00:36:00.647188 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:00.647174 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="481ba507-2925-4706-947c-898c75c7c210" containerName="revoke-keys" Nov 03 00:36:00.650442 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:00.650421 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2mxgc" Nov 03 00:36:00.704377 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:00.704299 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:36:00.704377 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:00.704318 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:36:00.704377 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:00.704346 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:36:00.704618 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:00.704389 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:36:00.708910 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:00.708885 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2mxgc"] Nov 03 00:36:00.795333 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:00.795234 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjz2n\" (UniqueName: \"kubernetes.io/projected/2f3f48f3-fc54-4b65-822c-7fafd58b03a6-kube-api-access-fjz2n\") pod \"maas-api-revoke-sub-3d4156f1-2mxgc\" (UID: \"2f3f48f3-fc54-4b65-822c-7fafd58b03a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2mxgc" Nov 03 00:36:00.795542 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:00.795337 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f3f48f3-fc54-4b65-822c-7fafd58b03a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2mxgc\" (UID: \"2f3f48f3-fc54-4b65-822c-7fafd58b03a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2mxgc" Nov 03 00:36:00.795542 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:00.795384 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f3f48f3-fc54-4b65-822c-7fafd58b03a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2mxgc\" (UID: \"2f3f48f3-fc54-4b65-822c-7fafd58b03a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2mxgc" Nov 03 00:36:00.896976 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:00.896943 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjz2n\" (UniqueName: \"kubernetes.io/projected/2f3f48f3-fc54-4b65-822c-7fafd58b03a6-kube-api-access-fjz2n\") pod \"maas-api-revoke-sub-3d4156f1-2mxgc\" (UID: \"2f3f48f3-fc54-4b65-822c-7fafd58b03a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2mxgc" Nov 03 00:36:00.897202 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:00.896984 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f3f48f3-fc54-4b65-822c-7fafd58b03a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2mxgc\" (UID: \"2f3f48f3-fc54-4b65-822c-7fafd58b03a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2mxgc" Nov 03 00:36:00.897202 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:00.897010 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f3f48f3-fc54-4b65-822c-7fafd58b03a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2mxgc\" (UID: \"2f3f48f3-fc54-4b65-822c-7fafd58b03a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2mxgc" Nov 03 00:36:00.897658 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:00.897636 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f3f48f3-fc54-4b65-822c-7fafd58b03a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2mxgc\" (UID: \"2f3f48f3-fc54-4b65-822c-7fafd58b03a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2mxgc" Nov 03 00:36:00.899710 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:00.899683 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f3f48f3-fc54-4b65-822c-7fafd58b03a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2mxgc\" (UID: \"2f3f48f3-fc54-4b65-822c-7fafd58b03a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2mxgc" Nov 03 00:36:00.957624 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:00.957592 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjz2n\" (UniqueName: \"kubernetes.io/projected/2f3f48f3-fc54-4b65-822c-7fafd58b03a6-kube-api-access-fjz2n\") pod \"maas-api-revoke-sub-3d4156f1-2mxgc\" (UID: \"2f3f48f3-fc54-4b65-822c-7fafd58b03a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2mxgc" Nov 03 00:36:00.963419 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:00.963393 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2mxgc" Nov 03 00:36:01.339753 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:01.339727 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2mxgc"] Nov 03 00:36:01.340882 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:36:01.340850 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f3f48f3_fc54_4b65_822c_7fafd58b03a6.slice/crio-fe09fd84b965139df0751aa3740ad85c5e1670a3d7622b376b5b75e8e68289e2 WatchSource:0}: Error finding container fe09fd84b965139df0751aa3740ad85c5e1670a3d7622b376b5b75e8e68289e2: Status 404 returned error can't find the container with id fe09fd84b965139df0751aa3740ad85c5e1670a3d7622b376b5b75e8e68289e2 Nov 03 00:36:01.355635 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:01.355608 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2mxgc" event={"ID":"2f3f48f3-fc54-4b65-822c-7fafd58b03a6","Type":"ContainerStarted","Data":"fe09fd84b965139df0751aa3740ad85c5e1670a3d7622b376b5b75e8e68289e2"} Nov 03 00:36:01.882197 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:01.882153 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="461b0b8f-3f8b-4efa-9de5-340f33232caf" path="/var/lib/kubelet/pods/461b0b8f-3f8b-4efa-9de5-340f33232caf/volumes" Nov 03 00:36:02.364575 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:02.364536 5727 generic.go:356] "Generic (PLEG): container finished" podID="2f3f48f3-fc54-4b65-822c-7fafd58b03a6" containerID="1bf5dc6c586169700708b228ab05e5cc235d4c02f9af0dc61d61669ef80e45f2" exitCode=0 Nov 03 00:36:02.364767 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:02.364589 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2mxgc" event={"ID":"2f3f48f3-fc54-4b65-822c-7fafd58b03a6","Type":"ContainerDied","Data":"1bf5dc6c586169700708b228ab05e5cc235d4c02f9af0dc61d61669ef80e45f2"} Nov 03 00:36:03.515620 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:03.515592 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2mxgc" Nov 03 00:36:03.624550 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:03.624479 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f3f48f3-fc54-4b65-822c-7fafd58b03a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f3f48f3-fc54-4b65-822c-7fafd58b03a6-maas-api-xks-ca\") pod \"2f3f48f3-fc54-4b65-822c-7fafd58b03a6\" (UID: \"2f3f48f3-fc54-4b65-822c-7fafd58b03a6\") " Nov 03 00:36:03.624550 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:03.624535 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f3f48f3-fc54-4b65-822c-7fafd58b03a6-kube-api-access-fjz2n\" (UniqueName: \"kubernetes.io/projected/2f3f48f3-fc54-4b65-822c-7fafd58b03a6-kube-api-access-fjz2n\") pod \"2f3f48f3-fc54-4b65-822c-7fafd58b03a6\" (UID: \"2f3f48f3-fc54-4b65-822c-7fafd58b03a6\") " Nov 03 00:36:03.624726 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:03.624560 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f3f48f3-fc54-4b65-822c-7fafd58b03a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f3f48f3-fc54-4b65-822c-7fafd58b03a6-maas-api-openshift-ca\") pod \"2f3f48f3-fc54-4b65-822c-7fafd58b03a6\" (UID: \"2f3f48f3-fc54-4b65-822c-7fafd58b03a6\") " Nov 03 00:36:03.625000 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:03.624973 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f3f48f3-fc54-4b65-822c-7fafd58b03a6-maas-api-openshift-ca" pod "2f3f48f3-fc54-4b65-822c-7fafd58b03a6" (UID: "2f3f48f3-fc54-4b65-822c-7fafd58b03a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:36:03.626720 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:03.626685 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f3f48f3-fc54-4b65-822c-7fafd58b03a6-kube-api-access-fjz2n" pod "2f3f48f3-fc54-4b65-822c-7fafd58b03a6" (UID: "2f3f48f3-fc54-4b65-822c-7fafd58b03a6"). InnerVolumeSpecName "kube-api-access-fjz2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:36:03.626829 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:03.626697 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f3f48f3-fc54-4b65-822c-7fafd58b03a6-maas-api-xks-ca" pod "2f3f48f3-fc54-4b65-822c-7fafd58b03a6" (UID: "2f3f48f3-fc54-4b65-822c-7fafd58b03a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:36:03.726367 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:03.726337 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f3f48f3-fc54-4b65-822c-7fafd58b03a6-maas-api-xks-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:03.726367 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:03.726365 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjz2n\" (UniqueName: \"kubernetes.io/projected/2f3f48f3-fc54-4b65-822c-7fafd58b03a6-kube-api-access-fjz2n\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:03.726367 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:03.726377 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f3f48f3-fc54-4b65-822c-7fafd58b03a6-maas-api-openshift-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:04.380874 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:04.380836 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2mxgc" Nov 03 00:36:04.381185 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:04.380876 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2mxgc" event={"ID":"2f3f48f3-fc54-4b65-822c-7fafd58b03a6","Type":"ContainerDied","Data":"fe09fd84b965139df0751aa3740ad85c5e1670a3d7622b376b5b75e8e68289e2"} Nov 03 00:36:04.381185 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:04.380947 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe09fd84b965139df0751aa3740ad85c5e1670a3d7622b376b5b75e8e68289e2" Nov 03 00:36:20.047372 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:20.047335 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qjhsz"] Nov 03 00:36:20.097477 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:20.097445 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qjhsz"] Nov 03 00:36:21.882450 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:21.882409 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="468a0873-51e8-4b7f-bf54-5131fd0352da" path="/var/lib/kubelet/pods/468a0873-51e8-4b7f-bf54-5131fd0352da/volumes" Nov 03 00:36:25.572456 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:25.572418 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-649f61-696578shkd"] Nov 03 00:36:25.573102 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:25.573081 5727 cpu_manager.go:414] "containerMap: removing container" podUID="2f3f48f3-fc54-4b65-822c-7fafd58b03a6" containerName="revoke-keys" Nov 03 00:36:25.573226 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:25.573104 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f3f48f3-fc54-4b65-822c-7fafd58b03a6" containerName="revoke-keys" Nov 03 00:36:25.573289 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:25.573276 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f3f48f3-fc54-4b65-822c-7fafd58b03a6" containerName="revoke-keys" Nov 03 00:36:25.577834 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:25.577812 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-649f61-696578shkd" Nov 03 00:36:25.646480 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:25.646445 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-649f61-696578shkd"] Nov 03 00:36:25.663890 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:25.663861 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdr79\" (UniqueName: \"kubernetes.io/projected/643db636-68df-4fa0-8977-258232f30fdc-kube-api-access-mdr79\") pod \"payload-pre-processing-e2e-worker-single-main-649f61-696578shkd\" (UID: \"643db636-68df-4fa0-8977-258232f30fdc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-649f61-696578shkd" Nov 03 00:36:25.664025 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:25.664006 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/643db636-68df-4fa0-8977-258232f30fdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-649f61-696578shkd\" (UID: \"643db636-68df-4fa0-8977-258232f30fdc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-649f61-696578shkd" Nov 03 00:36:25.679241 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:25.679221 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-649f61\"" Nov 03 00:36:25.729217 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:25.729182 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-649f61-795c44df9rdxwq"] Nov 03 00:36:25.733178 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:25.733155 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-649f61-795c44df9rdxwq" Nov 03 00:36:25.764735 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:25.764709 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdr79\" (UniqueName: \"kubernetes.io/projected/643db636-68df-4fa0-8977-258232f30fdc-kube-api-access-mdr79\") pod \"payload-pre-processing-e2e-worker-single-main-649f61-696578shkd\" (UID: \"643db636-68df-4fa0-8977-258232f30fdc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-649f61-696578shkd" Nov 03 00:36:25.764892 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:25.764809 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/643db636-68df-4fa0-8977-258232f30fdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-649f61-696578shkd\" (UID: \"643db636-68df-4fa0-8977-258232f30fdc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-649f61-696578shkd" Nov 03 00:36:25.765570 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:25.765548 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/643db636-68df-4fa0-8977-258232f30fdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-649f61-696578shkd\" (UID: \"643db636-68df-4fa0-8977-258232f30fdc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-649f61-696578shkd" Nov 03 00:36:25.785507 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:25.785481 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-63185c35-dockercfg-7j8px\"" Nov 03 00:36:25.789995 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:25.789974 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-649f61-795c44df9rdxwq"] Nov 03 00:36:25.826051 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:25.825987 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdr79\" (UniqueName: \"kubernetes.io/projected/643db636-68df-4fa0-8977-258232f30fdc-kube-api-access-mdr79\") pod \"payload-pre-processing-e2e-worker-single-main-649f61-696578shkd\" (UID: \"643db636-68df-4fa0-8977-258232f30fdc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-649f61-696578shkd" Nov 03 00:36:25.866278 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:25.866246 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50b01cb5-7a37-44c9-871a-0f213fe32f3e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-649f61-795c44df9rdxwq\" (UID: \"50b01cb5-7a37-44c9-871a-0f213fe32f3e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-649f61-795c44df9rdxwq" Nov 03 00:36:25.866441 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:25.866300 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt5x2\" (UniqueName: \"kubernetes.io/projected/50b01cb5-7a37-44c9-871a-0f213fe32f3e-kube-api-access-rt5x2\") pod \"payload-processing-e2e-worker-single-main-649f61-795c44df9rdxwq\" (UID: \"50b01cb5-7a37-44c9-871a-0f213fe32f3e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-649f61-795c44df9rdxwq" Nov 03 00:36:25.891445 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:25.891410 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-649f61-696578shkd" Nov 03 00:36:25.967796 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:25.967765 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50b01cb5-7a37-44c9-871a-0f213fe32f3e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-649f61-795c44df9rdxwq\" (UID: \"50b01cb5-7a37-44c9-871a-0f213fe32f3e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-649f61-795c44df9rdxwq" Nov 03 00:36:25.967933 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:25.967836 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt5x2\" (UniqueName: \"kubernetes.io/projected/50b01cb5-7a37-44c9-871a-0f213fe32f3e-kube-api-access-rt5x2\") pod \"payload-processing-e2e-worker-single-main-649f61-795c44df9rdxwq\" (UID: \"50b01cb5-7a37-44c9-871a-0f213fe32f3e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-649f61-795c44df9rdxwq" Nov 03 00:36:25.968969 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:25.968946 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50b01cb5-7a37-44c9-871a-0f213fe32f3e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-649f61-795c44df9rdxwq\" (UID: \"50b01cb5-7a37-44c9-871a-0f213fe32f3e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-649f61-795c44df9rdxwq" Nov 03 00:36:26.029483 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:26.029449 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt5x2\" (UniqueName: \"kubernetes.io/projected/50b01cb5-7a37-44c9-871a-0f213fe32f3e-kube-api-access-rt5x2\") pod \"payload-processing-e2e-worker-single-main-649f61-795c44df9rdxwq\" (UID: \"50b01cb5-7a37-44c9-871a-0f213fe32f3e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-649f61-795c44df9rdxwq" Nov 03 00:36:26.047596 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:26.047564 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-649f61-795c44df9rdxwq" Nov 03 00:36:26.269021 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:26.268997 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-649f61-696578shkd"] Nov 03 00:36:26.270768 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:36:26.270736 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod643db636_68df_4fa0_8977_258232f30fdc.slice/crio-2e59ed76488c7628d1253a0f888fe11548d79cf5a1141f482fabdfae1d6c1780 WatchSource:0}: Error finding container 2e59ed76488c7628d1253a0f888fe11548d79cf5a1141f482fabdfae1d6c1780: Status 404 returned error can't find the container with id 2e59ed76488c7628d1253a0f888fe11548d79cf5a1141f482fabdfae1d6c1780 Nov 03 00:36:26.425772 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:26.425741 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-649f61-795c44df9rdxwq"] Nov 03 00:36:26.426953 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:36:26.426925 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50b01cb5_7a37_44c9_871a_0f213fe32f3e.slice/crio-b211ab53ef364da8b8355cf199eb99841b5d8a9d47bb9d1376d30b413f38a338 WatchSource:0}: Error finding container b211ab53ef364da8b8355cf199eb99841b5d8a9d47bb9d1376d30b413f38a338: Status 404 returned error can't find the container with id b211ab53ef364da8b8355cf199eb99841b5d8a9d47bb9d1376d30b413f38a338 Nov 03 00:36:26.613667 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:26.613630 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-649f61-795c44df9rdxwq" event={"ID":"50b01cb5-7a37-44c9-871a-0f213fe32f3e","Type":"ContainerStarted","Data":"1c150d3543602e8dc05024d01730d5a4eaea4d637705e2cc6a74ac314fe5ee00"} Nov 03 00:36:26.614152 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:26.613686 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-649f61-795c44df9rdxwq" event={"ID":"50b01cb5-7a37-44c9-871a-0f213fe32f3e","Type":"ContainerStarted","Data":"b211ab53ef364da8b8355cf199eb99841b5d8a9d47bb9d1376d30b413f38a338"} Nov 03 00:36:26.614152 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:26.613742 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-649f61-795c44df9rdxwq" Nov 03 00:36:26.615563 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:26.615530 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-649f61-696578shkd" event={"ID":"643db636-68df-4fa0-8977-258232f30fdc","Type":"ContainerStarted","Data":"c75004d19d142aed6da204f7606133f4432386276c16ae953648b2d7f0bdb477"} Nov 03 00:36:26.616046 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:26.615576 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-649f61-696578shkd" event={"ID":"643db636-68df-4fa0-8977-258232f30fdc","Type":"ContainerStarted","Data":"2e59ed76488c7628d1253a0f888fe11548d79cf5a1141f482fabdfae1d6c1780"} Nov 03 00:36:26.616046 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:26.615643 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-649f61-696578shkd" Nov 03 00:36:26.730643 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:26.730541 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-649f61-795c44df9rdxwq" podStartSLOduration=1.730525889 podStartE2EDuration="1.730525889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:36:26.728892475 +0000 UTC m=+6472.967390137" watchObservedRunningTime="2026-11-03 00:36:26.730525889 +0000 UTC m=+6472.969023552" Nov 03 00:36:27.846007 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:36:27.845964 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:36:28.295679 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:28.295578 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-649f61-696578shkd" podStartSLOduration=3.295565011 podStartE2EDuration="3.295565011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:36:26.847334123 +0000 UTC m=+6473.085831785" watchObservedRunningTime="2026-11-03 00:36:28.295565011 +0000 UTC m=+6474.534062673" Nov 03 00:36:28.297987 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:28.297959 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc"] Nov 03 00:36:28.301913 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:28.301892 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc" Nov 03 00:36:28.390901 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:28.390866 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e2758e8-108d-4b10-9874-6ab1d4b6d8b2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc\" (UID: \"6e2758e8-108d-4b10-9874-6ab1d4b6d8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc" Nov 03 00:36:28.391064 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:28.390913 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e2758e8-108d-4b10-9874-6ab1d4b6d8b2-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc\" (UID: \"6e2758e8-108d-4b10-9874-6ab1d4b6d8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc" Nov 03 00:36:28.391064 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:28.390936 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e2758e8-108d-4b10-9874-6ab1d4b6d8b2-metrics-certs\") pod \"maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc\" (UID: \"6e2758e8-108d-4b10-9874-6ab1d4b6d8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc" Nov 03 00:36:28.391156 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:28.391093 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s5qh\" (UniqueName: \"kubernetes.io/projected/6e2758e8-108d-4b10-9874-6ab1d4b6d8b2-kube-api-access-2s5qh\") pod \"maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc\" (UID: \"6e2758e8-108d-4b10-9874-6ab1d4b6d8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc" Nov 03 00:36:28.406486 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:28.406454 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:36:28.406686 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:28.406500 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-649f61-serving-cert\"" Nov 03 00:36:28.406686 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:28.406509 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:36:28.406686 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:28.406539 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 00:36:28.406686 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:28.406575 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wm4gz\"" Nov 03 00:36:28.409713 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:28.409687 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc"] Nov 03 00:36:28.492206 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:28.492167 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e2758e8-108d-4b10-9874-6ab1d4b6d8b2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc\" (UID: \"6e2758e8-108d-4b10-9874-6ab1d4b6d8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc" Nov 03 00:36:28.492418 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:28.492237 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e2758e8-108d-4b10-9874-6ab1d4b6d8b2-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc\" (UID: \"6e2758e8-108d-4b10-9874-6ab1d4b6d8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc" Nov 03 00:36:28.492418 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:28.492263 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e2758e8-108d-4b10-9874-6ab1d4b6d8b2-metrics-certs\") pod \"maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc\" (UID: \"6e2758e8-108d-4b10-9874-6ab1d4b6d8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc" Nov 03 00:36:28.492418 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:36:28.492302 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-serving-cert: secret "maas-api-e2e-worker-single-main-649f61-serving-cert" not found Nov 03 00:36:28.492418 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:28.492334 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s5qh\" (UniqueName: \"kubernetes.io/projected/6e2758e8-108d-4b10-9874-6ab1d4b6d8b2-kube-api-access-2s5qh\") pod \"maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc\" (UID: \"6e2758e8-108d-4b10-9874-6ab1d4b6d8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc" Nov 03 00:36:28.492418 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:36:28.492365 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e2758e8-108d-4b10-9874-6ab1d4b6d8b2-maas-api-tls podName:6e2758e8-108d-4b10-9874-6ab1d4b6d8b2 nodeName:}" failed. No retries permitted until 2026-11-03 00:36:28.992348134 +0000 UTC m=+6475.230845774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6e2758e8-108d-4b10-9874-6ab1d4b6d8b2-maas-api-tls") pod "maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc" (UID: "6e2758e8-108d-4b10-9874-6ab1d4b6d8b2") : secret "maas-api-e2e-worker-single-main-649f61-serving-cert" not found Nov 03 00:36:28.493077 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:28.493054 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e2758e8-108d-4b10-9874-6ab1d4b6d8b2-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc\" (UID: \"6e2758e8-108d-4b10-9874-6ab1d4b6d8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc" Nov 03 00:36:28.495024 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:28.494998 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e2758e8-108d-4b10-9874-6ab1d4b6d8b2-metrics-certs\") pod \"maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc\" (UID: \"6e2758e8-108d-4b10-9874-6ab1d4b6d8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc" Nov 03 00:36:28.558927 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:28.558837 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s5qh\" (UniqueName: \"kubernetes.io/projected/6e2758e8-108d-4b10-9874-6ab1d4b6d8b2-kube-api-access-2s5qh\") pod \"maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc\" (UID: \"6e2758e8-108d-4b10-9874-6ab1d4b6d8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc" Nov 03 00:36:28.997196 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:28.997156 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e2758e8-108d-4b10-9874-6ab1d4b6d8b2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc\" (UID: \"6e2758e8-108d-4b10-9874-6ab1d4b6d8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc" Nov 03 00:36:28.999831 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:28.999804 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e2758e8-108d-4b10-9874-6ab1d4b6d8b2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc\" (UID: \"6e2758e8-108d-4b10-9874-6ab1d4b6d8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc" Nov 03 00:36:29.216704 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:29.216675 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc" Nov 03 00:36:29.594784 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:29.594756 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc"] Nov 03 00:36:29.596328 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:36:29.596292 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e2758e8_108d_4b10_9874_6ab1d4b6d8b2.slice/crio-7870d5089a0f9ba1a60386f84e0395910a5576ee0a1d7edf83f99146d92b6da6 WatchSource:0}: Error finding container 7870d5089a0f9ba1a60386f84e0395910a5576ee0a1d7edf83f99146d92b6da6: Status 404 returned error can't find the container with id 7870d5089a0f9ba1a60386f84e0395910a5576ee0a1d7edf83f99146d92b6da6 Nov 03 00:36:29.597886 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:29.597865 5727 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:36:29.637122 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:29.637086 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc" event={"ID":"6e2758e8-108d-4b10-9874-6ab1d4b6d8b2","Type":"ContainerStarted","Data":"7870d5089a0f9ba1a60386f84e0395910a5576ee0a1d7edf83f99146d92b6da6"} Nov 03 00:36:30.645801 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:30.645748 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc" event={"ID":"6e2758e8-108d-4b10-9874-6ab1d4b6d8b2","Type":"ContainerStarted","Data":"23a6cb3659344e3ba9673008081162ffa44158e4c3b6993943165f7919550a12"} Nov 03 00:36:30.646240 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:30.645902 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc" Nov 03 00:36:30.765025 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:30.764915 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc" podStartSLOduration=1.891240817 podStartE2EDuration="2.764901531s" totalImagesPullingTime="873.660714ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:36:30.762275931 +0000 UTC m=+6477.000773606" watchObservedRunningTime="2026-11-03 00:36:30.764901531 +0000 UTC m=+6477.003399193" Nov 03 00:36:36.657883 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:36.657855 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc" Nov 03 00:36:37.625655 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:37.625613 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-649f61-696578shkd" Nov 03 00:36:37.625849 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:37.625694 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-649f61-795c44df9rdxwq" Nov 03 00:36:45.048060 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:45.047979 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z424d"] Nov 03 00:36:45.098835 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:45.098798 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z424d"] Nov 03 00:36:45.882552 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:45.882520 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5624412f-4ebe-40ad-835c-0df126ea4729" path="/var/lib/kubelet/pods/5624412f-4ebe-40ad-835c-0df126ea4729/volumes" Nov 03 00:36:57.046852 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:57.046816 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bk6rr"] Nov 03 00:36:57.096603 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:57.096564 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bk6rr"] Nov 03 00:36:57.846942 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:36:57.846904 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:36:57.881493 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:36:57.881463 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80cd4ece-bcb6-4c3f-a303-b5d3100d638c" path="/var/lib/kubelet/pods/80cd4ece-bcb6-4c3f-a303-b5d3100d638c/volumes" Nov 03 00:37:27.845679 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:37:27.845631 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:37:32.046908 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:32.046876 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7rqm7"] Nov 03 00:37:32.095865 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:32.095830 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7rqm7"] Nov 03 00:37:33.883075 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:33.883045 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c1586c0-c0e3-47e3-a0c4-132e66b50514" path="/var/lib/kubelet/pods/7c1586c0-c0e3-47e3-a0c4-132e66b50514/volumes" Nov 03 00:37:36.472987 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:36.472953 5727 scope.go:122] "RemoveContainer" containerID="8572dfc9be53e189cfce50d3ded3f846451e4dc8d1f35e3b14dda3bc3e1d9c1d" Nov 03 00:37:36.483342 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:36.483320 5727 scope.go:122] "RemoveContainer" containerID="8f604c17966c27d8c4e8c75cfbc384a6e7904b44de9274b71c2a09fa118a27a7" Nov 03 00:37:43.047317 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:43.047275 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtppm"] Nov 03 00:37:43.096466 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:43.096433 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtppm"] Nov 03 00:37:43.882843 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:43.882798 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23db4f9f-8f48-4832-ad67-d9febdfb5633" path="/var/lib/kubelet/pods/23db4f9f-8f48-4832-ad67-d9febdfb5633/volumes" Nov 03 00:37:52.354927 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:52.354886 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-649f61-795c44df9rdxwq"] Nov 03 00:37:52.461076 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:52.461044 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-649f61-696578shkd"] Nov 03 00:37:57.355504 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:57.355465 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-649f61-795c44df9rdxwq" podUID="50b01cb5-7a37-44c9-871a-0f213fe32f3e" containerName="payload-processing" containerID="cri-o://1c150d3543602e8dc05024d01730d5a4eaea4d637705e2cc6a74ac314fe5ee00" gracePeriod=25 Nov 03 00:37:57.462291 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:57.462181 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-649f61-696578shkd" podUID="643db636-68df-4fa0-8977-258232f30fdc" containerName="payload-pre-processing" containerID="cri-o://c75004d19d142aed6da204f7606133f4432386276c16ae953648b2d7f0bdb477" gracePeriod=25 Nov 03 00:37:57.511453 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:57.511413 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc"] Nov 03 00:37:57.511653 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:57.511631 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc" podUID="6e2758e8-108d-4b10-9874-6ab1d4b6d8b2" containerName="maas-api" containerID="cri-o://23a6cb3659344e3ba9673008081162ffa44158e4c3b6993943165f7919550a12" gracePeriod=30 Nov 03 00:37:57.624251 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:57.624217 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-649f61-795c44df9rdxwq" podUID="50b01cb5-7a37-44c9-871a-0f213fe32f3e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:37:57.624445 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:57.624226 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-649f61-696578shkd" podUID="643db636-68df-4fa0-8977-258232f30fdc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:37:57.845194 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:37:57.845161 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:37:58.792772 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:58.792742 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc" Nov 03 00:37:58.881000 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:58.880913 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e2758e8-108d-4b10-9874-6ab1d4b6d8b2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e2758e8-108d-4b10-9874-6ab1d4b6d8b2-openshift-service-ca\") pod \"6e2758e8-108d-4b10-9874-6ab1d4b6d8b2\" (UID: \"6e2758e8-108d-4b10-9874-6ab1d4b6d8b2\") " Nov 03 00:37:58.881000 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:58.880972 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e2758e8-108d-4b10-9874-6ab1d4b6d8b2-kube-api-access-2s5qh\" (UniqueName: \"kubernetes.io/projected/6e2758e8-108d-4b10-9874-6ab1d4b6d8b2-kube-api-access-2s5qh\") pod \"6e2758e8-108d-4b10-9874-6ab1d4b6d8b2\" (UID: \"6e2758e8-108d-4b10-9874-6ab1d4b6d8b2\") " Nov 03 00:37:58.881297 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:58.881037 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e2758e8-108d-4b10-9874-6ab1d4b6d8b2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e2758e8-108d-4b10-9874-6ab1d4b6d8b2-metrics-certs\") pod \"6e2758e8-108d-4b10-9874-6ab1d4b6d8b2\" (UID: \"6e2758e8-108d-4b10-9874-6ab1d4b6d8b2\") " Nov 03 00:37:58.881297 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:58.881193 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e2758e8-108d-4b10-9874-6ab1d4b6d8b2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e2758e8-108d-4b10-9874-6ab1d4b6d8b2-maas-api-tls\") pod \"6e2758e8-108d-4b10-9874-6ab1d4b6d8b2\" (UID: \"6e2758e8-108d-4b10-9874-6ab1d4b6d8b2\") " Nov 03 00:37:58.881414 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:58.881373 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e2758e8-108d-4b10-9874-6ab1d4b6d8b2-openshift-service-ca" pod "6e2758e8-108d-4b10-9874-6ab1d4b6d8b2" (UID: "6e2758e8-108d-4b10-9874-6ab1d4b6d8b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:37:58.881644 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:58.881619 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e2758e8-108d-4b10-9874-6ab1d4b6d8b2-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:58.883409 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:58.883384 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e2758e8-108d-4b10-9874-6ab1d4b6d8b2-metrics-certs" pod "6e2758e8-108d-4b10-9874-6ab1d4b6d8b2" (UID: "6e2758e8-108d-4b10-9874-6ab1d4b6d8b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:37:58.883729 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:58.883711 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e2758e8-108d-4b10-9874-6ab1d4b6d8b2-kube-api-access-2s5qh" pod "6e2758e8-108d-4b10-9874-6ab1d4b6d8b2" (UID: "6e2758e8-108d-4b10-9874-6ab1d4b6d8b2"). InnerVolumeSpecName "kube-api-access-2s5qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:37:58.883883 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:58.883863 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e2758e8-108d-4b10-9874-6ab1d4b6d8b2-maas-api-tls" pod "6e2758e8-108d-4b10-9874-6ab1d4b6d8b2" (UID: "6e2758e8-108d-4b10-9874-6ab1d4b6d8b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:37:58.983051 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:58.983015 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e2758e8-108d-4b10-9874-6ab1d4b6d8b2-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:58.983051 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:58.983053 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s5qh\" (UniqueName: \"kubernetes.io/projected/6e2758e8-108d-4b10-9874-6ab1d4b6d8b2-kube-api-access-2s5qh\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:58.983330 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:58.983068 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e2758e8-108d-4b10-9874-6ab1d4b6d8b2-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:59.278255 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:59.278160 5727 generic.go:356] "Generic (PLEG): container finished" podID="6e2758e8-108d-4b10-9874-6ab1d4b6d8b2" containerID="23a6cb3659344e3ba9673008081162ffa44158e4c3b6993943165f7919550a12" exitCode=0 Nov 03 00:37:59.278255 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:59.278200 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc" Nov 03 00:37:59.278698 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:59.278252 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc" event={"ID":"6e2758e8-108d-4b10-9874-6ab1d4b6d8b2","Type":"ContainerDied","Data":"23a6cb3659344e3ba9673008081162ffa44158e4c3b6993943165f7919550a12"} Nov 03 00:37:59.278698 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:59.278307 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc" event={"ID":"6e2758e8-108d-4b10-9874-6ab1d4b6d8b2","Type":"ContainerDied","Data":"7870d5089a0f9ba1a60386f84e0395910a5576ee0a1d7edf83f99146d92b6da6"} Nov 03 00:37:59.278698 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:59.278332 5727 scope.go:122] "RemoveContainer" containerID="23a6cb3659344e3ba9673008081162ffa44158e4c3b6993943165f7919550a12" Nov 03 00:37:59.289632 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:59.289576 5727 scope.go:122] "RemoveContainer" containerID="23a6cb3659344e3ba9673008081162ffa44158e4c3b6993943165f7919550a12" Nov 03 00:37:59.289943 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:37:59.289922 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23a6cb3659344e3ba9673008081162ffa44158e4c3b6993943165f7919550a12\": container with ID starting with 23a6cb3659344e3ba9673008081162ffa44158e4c3b6993943165f7919550a12 not found: ID does not exist" containerID="23a6cb3659344e3ba9673008081162ffa44158e4c3b6993943165f7919550a12" Nov 03 00:37:59.290007 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:59.289951 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23a6cb3659344e3ba9673008081162ffa44158e4c3b6993943165f7919550a12"} err="failed to get container status \"23a6cb3659344e3ba9673008081162ffa44158e4c3b6993943165f7919550a12\": rpc error: code = NotFound desc = could not find container \"23a6cb3659344e3ba9673008081162ffa44158e4c3b6993943165f7919550a12\": container with ID starting with 23a6cb3659344e3ba9673008081162ffa44158e4c3b6993943165f7919550a12 not found: ID does not exist" Nov 03 00:37:59.442777 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:59.442742 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc"] Nov 03 00:37:59.492554 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:59.492525 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-54hmc"] Nov 03 00:37:59.882599 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:37:59.882552 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e2758e8-108d-4b10-9874-6ab1d4b6d8b2" path="/var/lib/kubelet/pods/6e2758e8-108d-4b10-9874-6ab1d4b6d8b2/volumes" Nov 03 00:38:07.624058 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:07.624023 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-649f61-795c44df9rdxwq" podUID="50b01cb5-7a37-44c9-871a-0f213fe32f3e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:38:07.624512 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:07.624021 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-649f61-696578shkd" podUID="643db636-68df-4fa0-8977-258232f30fdc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:38:17.621535 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:17.621448 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-649f61-795c44df9rdxwq" Nov 03 00:38:17.623741 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:38:17.623719 5727 logging.go:55] [core] [Channel #17581 SubChannel #17582]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.234:50052", ServerName: "10.132.0.234:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.234:50052: connect: connection refused" Nov 03 00:38:17.667708 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:17.667681 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50b01cb5-7a37-44c9-871a-0f213fe32f3e-kube-api-access-rt5x2\" (UniqueName: \"kubernetes.io/projected/50b01cb5-7a37-44c9-871a-0f213fe32f3e-kube-api-access-rt5x2\") pod \"50b01cb5-7a37-44c9-871a-0f213fe32f3e\" (UID: \"50b01cb5-7a37-44c9-871a-0f213fe32f3e\") " Nov 03 00:38:17.667880 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:17.667862 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50b01cb5-7a37-44c9-871a-0f213fe32f3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50b01cb5-7a37-44c9-871a-0f213fe32f3e-plugins-config-volume\") pod \"50b01cb5-7a37-44c9-871a-0f213fe32f3e\" (UID: \"50b01cb5-7a37-44c9-871a-0f213fe32f3e\") " Nov 03 00:38:17.668307 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:17.668268 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50b01cb5-7a37-44c9-871a-0f213fe32f3e-plugins-config-volume" pod "50b01cb5-7a37-44c9-871a-0f213fe32f3e" (UID: "50b01cb5-7a37-44c9-871a-0f213fe32f3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:38:17.670811 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:17.670782 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b01cb5-7a37-44c9-871a-0f213fe32f3e-kube-api-access-rt5x2" pod "50b01cb5-7a37-44c9-871a-0f213fe32f3e" (UID: "50b01cb5-7a37-44c9-871a-0f213fe32f3e"). InnerVolumeSpecName "kube-api-access-rt5x2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:38:17.726026 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:17.726004 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-649f61-696578shkd" Nov 03 00:38:17.769222 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:17.769191 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/643db636-68df-4fa0-8977-258232f30fdc-kube-api-access-mdr79\" (UniqueName: \"kubernetes.io/projected/643db636-68df-4fa0-8977-258232f30fdc-kube-api-access-mdr79\") pod \"643db636-68df-4fa0-8977-258232f30fdc\" (UID: \"643db636-68df-4fa0-8977-258232f30fdc\") " Nov 03 00:38:17.769395 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:17.769232 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/643db636-68df-4fa0-8977-258232f30fdc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/643db636-68df-4fa0-8977-258232f30fdc-plugins-config-volume\") pod \"643db636-68df-4fa0-8977-258232f30fdc\" (UID: \"643db636-68df-4fa0-8977-258232f30fdc\") " Nov 03 00:38:17.769540 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:17.769526 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50b01cb5-7a37-44c9-871a-0f213fe32f3e-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:17.769540 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:17.769540 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt5x2\" (UniqueName: \"kubernetes.io/projected/50b01cb5-7a37-44c9-871a-0f213fe32f3e-kube-api-access-rt5x2\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:17.769725 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:17.769661 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/643db636-68df-4fa0-8977-258232f30fdc-plugins-config-volume" pod "643db636-68df-4fa0-8977-258232f30fdc" (UID: "643db636-68df-4fa0-8977-258232f30fdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:38:17.771483 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:17.771462 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/643db636-68df-4fa0-8977-258232f30fdc-kube-api-access-mdr79" pod "643db636-68df-4fa0-8977-258232f30fdc" (UID: "643db636-68df-4fa0-8977-258232f30fdc"). InnerVolumeSpecName "kube-api-access-mdr79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:38:17.870585 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:17.870557 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdr79\" (UniqueName: \"kubernetes.io/projected/643db636-68df-4fa0-8977-258232f30fdc-kube-api-access-mdr79\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:17.870585 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:17.870582 5727 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/643db636-68df-4fa0-8977-258232f30fdc-plugins-config-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:18.403548 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:18.403516 5727 generic.go:356] "Generic (PLEG): container finished" podID="50b01cb5-7a37-44c9-871a-0f213fe32f3e" containerID="1c150d3543602e8dc05024d01730d5a4eaea4d637705e2cc6a74ac314fe5ee00" exitCode=0 Nov 03 00:38:18.403741 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:18.403564 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-649f61-795c44df9rdxwq" Nov 03 00:38:18.403741 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:18.403596 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-649f61-795c44df9rdxwq" event={"ID":"50b01cb5-7a37-44c9-871a-0f213fe32f3e","Type":"ContainerDied","Data":"1c150d3543602e8dc05024d01730d5a4eaea4d637705e2cc6a74ac314fe5ee00"} Nov 03 00:38:18.403741 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:18.403650 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-649f61-795c44df9rdxwq" event={"ID":"50b01cb5-7a37-44c9-871a-0f213fe32f3e","Type":"ContainerDied","Data":"b211ab53ef364da8b8355cf199eb99841b5d8a9d47bb9d1376d30b413f38a338"} Nov 03 00:38:18.403741 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:18.403674 5727 scope.go:122] "RemoveContainer" containerID="1c150d3543602e8dc05024d01730d5a4eaea4d637705e2cc6a74ac314fe5ee00" Nov 03 00:38:18.405246 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:18.405216 5727 generic.go:356] "Generic (PLEG): container finished" podID="643db636-68df-4fa0-8977-258232f30fdc" containerID="c75004d19d142aed6da204f7606133f4432386276c16ae953648b2d7f0bdb477" exitCode=0 Nov 03 00:38:18.405389 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:18.405320 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-649f61-696578shkd" event={"ID":"643db636-68df-4fa0-8977-258232f30fdc","Type":"ContainerDied","Data":"c75004d19d142aed6da204f7606133f4432386276c16ae953648b2d7f0bdb477"} Nov 03 00:38:18.405389 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:18.405368 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-649f61-696578shkd" event={"ID":"643db636-68df-4fa0-8977-258232f30fdc","Type":"ContainerDied","Data":"2e59ed76488c7628d1253a0f888fe11548d79cf5a1141f482fabdfae1d6c1780"} Nov 03 00:38:18.405496 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:18.405436 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-649f61-696578shkd" Nov 03 00:38:18.413812 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:18.413796 5727 scope.go:122] "RemoveContainer" containerID="1c150d3543602e8dc05024d01730d5a4eaea4d637705e2cc6a74ac314fe5ee00" Nov 03 00:38:18.414101 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:38:18.414081 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c150d3543602e8dc05024d01730d5a4eaea4d637705e2cc6a74ac314fe5ee00\": container with ID starting with 1c150d3543602e8dc05024d01730d5a4eaea4d637705e2cc6a74ac314fe5ee00 not found: ID does not exist" containerID="1c150d3543602e8dc05024d01730d5a4eaea4d637705e2cc6a74ac314fe5ee00" Nov 03 00:38:18.414174 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:18.414110 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c150d3543602e8dc05024d01730d5a4eaea4d637705e2cc6a74ac314fe5ee00"} err="failed to get container status \"1c150d3543602e8dc05024d01730d5a4eaea4d637705e2cc6a74ac314fe5ee00\": rpc error: code = NotFound desc = could not find container \"1c150d3543602e8dc05024d01730d5a4eaea4d637705e2cc6a74ac314fe5ee00\": container with ID starting with 1c150d3543602e8dc05024d01730d5a4eaea4d637705e2cc6a74ac314fe5ee00 not found: ID does not exist" Nov 03 00:38:18.414174 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:18.414155 5727 scope.go:122] "RemoveContainer" containerID="c75004d19d142aed6da204f7606133f4432386276c16ae953648b2d7f0bdb477" Nov 03 00:38:18.422933 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:18.422916 5727 scope.go:122] "RemoveContainer" containerID="c75004d19d142aed6da204f7606133f4432386276c16ae953648b2d7f0bdb477" Nov 03 00:38:18.423286 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:38:18.423267 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c75004d19d142aed6da204f7606133f4432386276c16ae953648b2d7f0bdb477\": container with ID starting with c75004d19d142aed6da204f7606133f4432386276c16ae953648b2d7f0bdb477 not found: ID does not exist" containerID="c75004d19d142aed6da204f7606133f4432386276c16ae953648b2d7f0bdb477" Nov 03 00:38:18.423345 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:18.423292 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c75004d19d142aed6da204f7606133f4432386276c16ae953648b2d7f0bdb477"} err="failed to get container status \"c75004d19d142aed6da204f7606133f4432386276c16ae953648b2d7f0bdb477\": rpc error: code = NotFound desc = could not find container \"c75004d19d142aed6da204f7606133f4432386276c16ae953648b2d7f0bdb477\": container with ID starting with c75004d19d142aed6da204f7606133f4432386276c16ae953648b2d7f0bdb477 not found: ID does not exist" Nov 03 00:38:18.566851 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:18.566816 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-649f61-696578shkd"] Nov 03 00:38:18.616182 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:18.616153 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-649f61-696578shkd"] Nov 03 00:38:18.624097 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:18.624047 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-649f61-696578shkd" podUID="643db636-68df-4fa0-8977-258232f30fdc" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.234:50052\" within 1s: context deadline exceeded" Nov 03 00:38:18.680582 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:18.680555 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-649f61-795c44df9rdxwq"] Nov 03 00:38:18.685755 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:18.685729 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-649f61-795c44df9rdxwq"] Nov 03 00:38:19.047944 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:19.047862 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hkq8r"] Nov 03 00:38:19.050422 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:19.050401 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hkq8r"] Nov 03 00:38:19.881884 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:19.881847 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="053fafe7-6bb7-439b-91c7-f33dc833cae2" path="/var/lib/kubelet/pods/053fafe7-6bb7-439b-91c7-f33dc833cae2/volumes" Nov 03 00:38:19.882285 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:19.882270 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50b01cb5-7a37-44c9-871a-0f213fe32f3e" path="/var/lib/kubelet/pods/50b01cb5-7a37-44c9-871a-0f213fe32f3e/volumes" Nov 03 00:38:19.882608 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:19.882595 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="643db636-68df-4fa0-8977-258232f30fdc" path="/var/lib/kubelet/pods/643db636-68df-4fa0-8977-258232f30fdc/volumes" Nov 03 00:38:27.846447 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:38:27.846409 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:38:31.503723 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:31.503692 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh"] Nov 03 00:38:31.504398 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:31.504376 5727 cpu_manager.go:414] "containerMap: removing container" podUID="643db636-68df-4fa0-8977-258232f30fdc" containerName="payload-pre-processing" Nov 03 00:38:31.504398 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:31.504394 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="643db636-68df-4fa0-8977-258232f30fdc" containerName="payload-pre-processing" Nov 03 00:38:31.504526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:31.504450 5727 cpu_manager.go:414] "containerMap: removing container" podUID="6e2758e8-108d-4b10-9874-6ab1d4b6d8b2" containerName="maas-api" Nov 03 00:38:31.504526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:31.504459 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e2758e8-108d-4b10-9874-6ab1d4b6d8b2" containerName="maas-api" Nov 03 00:38:31.504526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:31.504470 5727 cpu_manager.go:414] "containerMap: removing container" podUID="50b01cb5-7a37-44c9-871a-0f213fe32f3e" containerName="payload-processing" Nov 03 00:38:31.504526 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:31.504477 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50b01cb5-7a37-44c9-871a-0f213fe32f3e" containerName="payload-processing" Nov 03 00:38:31.504731 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:31.504583 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e2758e8-108d-4b10-9874-6ab1d4b6d8b2" containerName="maas-api" Nov 03 00:38:31.504731 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:31.504596 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="50b01cb5-7a37-44c9-871a-0f213fe32f3e" containerName="payload-processing" Nov 03 00:38:31.504731 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:31.504609 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="643db636-68df-4fa0-8977-258232f30fdc" containerName="payload-pre-processing" Nov 03 00:38:31.515206 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:31.515182 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh" Nov 03 00:38:31.564991 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:31.564960 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh"] Nov 03 00:38:31.567038 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:31.567013 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-649f61-serving-cert\"" Nov 03 00:38:31.598567 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:31.598533 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a98fb70-3ec5-4999-ba43-fa137d3cca2b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh\" (UID: \"9a98fb70-3ec5-4999-ba43-fa137d3cca2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh" Nov 03 00:38:31.598743 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:31.598581 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a98fb70-3ec5-4999-ba43-fa137d3cca2b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh\" (UID: \"9a98fb70-3ec5-4999-ba43-fa137d3cca2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh" Nov 03 00:38:31.598743 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:31.598690 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a98fb70-3ec5-4999-ba43-fa137d3cca2b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh\" (UID: \"9a98fb70-3ec5-4999-ba43-fa137d3cca2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh" Nov 03 00:38:31.598743 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:31.598721 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bxjv\" (UniqueName: \"kubernetes.io/projected/9a98fb70-3ec5-4999-ba43-fa137d3cca2b-kube-api-access-2bxjv\") pod \"maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh\" (UID: \"9a98fb70-3ec5-4999-ba43-fa137d3cca2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh" Nov 03 00:38:31.610845 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:31.610813 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:38:31.610845 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:31.610816 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 00:38:31.611054 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:31.610861 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:38:31.616595 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:31.616572 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wm4gz\"" Nov 03 00:38:31.699434 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:31.699396 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a98fb70-3ec5-4999-ba43-fa137d3cca2b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh\" (UID: \"9a98fb70-3ec5-4999-ba43-fa137d3cca2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh" Nov 03 00:38:31.699642 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:31.699483 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a98fb70-3ec5-4999-ba43-fa137d3cca2b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh\" (UID: \"9a98fb70-3ec5-4999-ba43-fa137d3cca2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh" Nov 03 00:38:31.699642 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:31.699554 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a98fb70-3ec5-4999-ba43-fa137d3cca2b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh\" (UID: \"9a98fb70-3ec5-4999-ba43-fa137d3cca2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh" Nov 03 00:38:31.699642 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:31.699582 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bxjv\" (UniqueName: \"kubernetes.io/projected/9a98fb70-3ec5-4999-ba43-fa137d3cca2b-kube-api-access-2bxjv\") pod \"maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh\" (UID: \"9a98fb70-3ec5-4999-ba43-fa137d3cca2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh" Nov 03 00:38:31.699993 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:38:31.699966 5727 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-serving-cert: secret "maas-api-e2e-worker-single-main-649f61-serving-cert" not found Nov 03 00:38:31.700182 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:38:31.700048 5727 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a98fb70-3ec5-4999-ba43-fa137d3cca2b-maas-api-tls podName:9a98fb70-3ec5-4999-ba43-fa137d3cca2b nodeName:}" failed. No retries permitted until 2026-11-03 00:38:32.200027983 +0000 UTC m=+6598.438525636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9a98fb70-3ec5-4999-ba43-fa137d3cca2b-maas-api-tls") pod "maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh" (UID: "9a98fb70-3ec5-4999-ba43-fa137d3cca2b") : secret "maas-api-e2e-worker-single-main-649f61-serving-cert" not found Nov 03 00:38:31.700279 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:31.700226 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a98fb70-3ec5-4999-ba43-fa137d3cca2b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh\" (UID: \"9a98fb70-3ec5-4999-ba43-fa137d3cca2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh" Nov 03 00:38:31.702536 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:31.702509 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a98fb70-3ec5-4999-ba43-fa137d3cca2b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh\" (UID: \"9a98fb70-3ec5-4999-ba43-fa137d3cca2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh" Nov 03 00:38:31.760991 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:31.760915 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bxjv\" (UniqueName: \"kubernetes.io/projected/9a98fb70-3ec5-4999-ba43-fa137d3cca2b-kube-api-access-2bxjv\") pod \"maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh\" (UID: \"9a98fb70-3ec5-4999-ba43-fa137d3cca2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh" Nov 03 00:38:32.204229 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:32.204188 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a98fb70-3ec5-4999-ba43-fa137d3cca2b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh\" (UID: \"9a98fb70-3ec5-4999-ba43-fa137d3cca2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh" Nov 03 00:38:32.206907 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:32.206875 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a98fb70-3ec5-4999-ba43-fa137d3cca2b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh\" (UID: \"9a98fb70-3ec5-4999-ba43-fa137d3cca2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh" Nov 03 00:38:32.430298 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:32.430261 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh" Nov 03 00:38:32.812708 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:32.812678 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh"] Nov 03 00:38:32.815951 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:38:32.815925 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a98fb70_3ec5_4999_ba43_fa137d3cca2b.slice/crio-cc2aba5dd2568252e6eb33e1d4633552592c157eb60cd65dec11fdc0bde7aa19 WatchSource:0}: Error finding container cc2aba5dd2568252e6eb33e1d4633552592c157eb60cd65dec11fdc0bde7aa19: Status 404 returned error can't find the container with id cc2aba5dd2568252e6eb33e1d4633552592c157eb60cd65dec11fdc0bde7aa19 Nov 03 00:38:33.527849 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:33.527807 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh" event={"ID":"9a98fb70-3ec5-4999-ba43-fa137d3cca2b","Type":"ContainerStarted","Data":"cc2aba5dd2568252e6eb33e1d4633552592c157eb60cd65dec11fdc0bde7aa19"} Nov 03 00:38:34.537152 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:34.537085 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh" event={"ID":"9a98fb70-3ec5-4999-ba43-fa137d3cca2b","Type":"ContainerStarted","Data":"14b2c2023385dca8b43217a96b4a857f5e4bedd4a9972ffd5879656bb34740bc"} Nov 03 00:38:34.537680 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:34.537178 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh" Nov 03 00:38:34.654787 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:34.654733 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh" podStartSLOduration=2.728316484 podStartE2EDuration="3.654720678s" totalImagesPullingTime="926.404194ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:38:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:38:34.65340684 +0000 UTC m=+6600.891904524" watchObservedRunningTime="2026-11-03 00:38:34.654720678 +0000 UTC m=+6600.893218339" Nov 03 00:38:35.280223 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:35.280190 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 03 00:38:35.297312 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:35.297287 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 03 00:38:35.378933 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:35.378905 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 03 00:38:35.394584 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:35.394558 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 03 00:38:36.516373 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:36.516342 5727 scope.go:122] "RemoveContainer" containerID="10cdf98dfeda9d4c38e0981e32f60df469c337103c5e1a1ff5c232b48d91017b" Nov 03 00:38:36.526917 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:36.526896 5727 scope.go:122] "RemoveContainer" containerID="afe9f65098b86fc6762c5e65fac00a00747b445e71b018d4a4a8b82d774c8966" Nov 03 00:38:36.540027 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:36.540004 5727 scope.go:122] "RemoveContainer" containerID="d459a28d6af98e395de959e5b0b5a2df389c900bf5e1d13cc14b810c804c8635" Nov 03 00:38:37.048415 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:37.048373 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2cjz"] Nov 03 00:38:37.096252 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:37.096214 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2cjz"] Nov 03 00:38:37.882199 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:37.882168 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c84aa5c3-904d-4577-99c3-48886c07a617" path="/var/lib/kubelet/pods/c84aa5c3-904d-4577-99c3-48886c07a617/volumes" Nov 03 00:38:40.549829 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:40.549802 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh" Nov 03 00:38:57.845659 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:38:57.845611 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:38:58.047569 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:58.047537 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v2ck7"] Nov 03 00:38:58.096846 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:58.096780 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v2ck7"] Nov 03 00:38:59.882078 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:38:59.882031 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42293b14-7074-4a8e-90f2-75ce0828dbc8" path="/var/lib/kubelet/pods/42293b14-7074-4a8e-90f2-75ce0828dbc8/volumes" Nov 03 00:39:00.489599 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:00.489569 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh"] Nov 03 00:39:00.489798 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:00.489772 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh" podUID="9a98fb70-3ec5-4999-ba43-fa137d3cca2b" containerName="maas-api" containerID="cri-o://14b2c2023385dca8b43217a96b4a857f5e4bedd4a9972ffd5879656bb34740bc" gracePeriod=30 Nov 03 00:39:00.545097 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:00.545053 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh" podUID="9a98fb70-3ec5-4999-ba43-fa137d3cca2b" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.237:8443/health\": dial tcp 10.132.0.237:8443: connect: connection refused" Nov 03 00:39:00.718443 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:00.718380 5727 generic.go:356] "Generic (PLEG): container finished" podID="9a98fb70-3ec5-4999-ba43-fa137d3cca2b" containerID="14b2c2023385dca8b43217a96b4a857f5e4bedd4a9972ffd5879656bb34740bc" exitCode=0 Nov 03 00:39:00.718718 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:00.718600 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh" event={"ID":"9a98fb70-3ec5-4999-ba43-fa137d3cca2b","Type":"ContainerDied","Data":"14b2c2023385dca8b43217a96b4a857f5e4bedd4a9972ffd5879656bb34740bc"} Nov 03 00:39:00.766281 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:00.766257 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh" Nov 03 00:39:00.796512 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:00.796488 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a98fb70-3ec5-4999-ba43-fa137d3cca2b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a98fb70-3ec5-4999-ba43-fa137d3cca2b-metrics-certs\") pod \"9a98fb70-3ec5-4999-ba43-fa137d3cca2b\" (UID: \"9a98fb70-3ec5-4999-ba43-fa137d3cca2b\") " Nov 03 00:39:00.796646 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:00.796530 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a98fb70-3ec5-4999-ba43-fa137d3cca2b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a98fb70-3ec5-4999-ba43-fa137d3cca2b-maas-api-tls\") pod \"9a98fb70-3ec5-4999-ba43-fa137d3cca2b\" (UID: \"9a98fb70-3ec5-4999-ba43-fa137d3cca2b\") " Nov 03 00:39:00.796646 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:00.796594 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a98fb70-3ec5-4999-ba43-fa137d3cca2b-kube-api-access-2bxjv\" (UniqueName: \"kubernetes.io/projected/9a98fb70-3ec5-4999-ba43-fa137d3cca2b-kube-api-access-2bxjv\") pod \"9a98fb70-3ec5-4999-ba43-fa137d3cca2b\" (UID: \"9a98fb70-3ec5-4999-ba43-fa137d3cca2b\") " Nov 03 00:39:00.796773 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:00.796655 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a98fb70-3ec5-4999-ba43-fa137d3cca2b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a98fb70-3ec5-4999-ba43-fa137d3cca2b-openshift-service-ca\") pod \"9a98fb70-3ec5-4999-ba43-fa137d3cca2b\" (UID: \"9a98fb70-3ec5-4999-ba43-fa137d3cca2b\") " Nov 03 00:39:00.797046 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:00.797016 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a98fb70-3ec5-4999-ba43-fa137d3cca2b-openshift-service-ca" pod "9a98fb70-3ec5-4999-ba43-fa137d3cca2b" (UID: "9a98fb70-3ec5-4999-ba43-fa137d3cca2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:39:00.798606 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:00.798582 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a98fb70-3ec5-4999-ba43-fa137d3cca2b-metrics-certs" pod "9a98fb70-3ec5-4999-ba43-fa137d3cca2b" (UID: "9a98fb70-3ec5-4999-ba43-fa137d3cca2b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:39:00.798776 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:00.798752 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a98fb70-3ec5-4999-ba43-fa137d3cca2b-maas-api-tls" pod "9a98fb70-3ec5-4999-ba43-fa137d3cca2b" (UID: "9a98fb70-3ec5-4999-ba43-fa137d3cca2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:39:00.798828 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:00.798778 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a98fb70-3ec5-4999-ba43-fa137d3cca2b-kube-api-access-2bxjv" pod "9a98fb70-3ec5-4999-ba43-fa137d3cca2b" (UID: "9a98fb70-3ec5-4999-ba43-fa137d3cca2b"). InnerVolumeSpecName "kube-api-access-2bxjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:39:00.898520 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:00.898477 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a98fb70-3ec5-4999-ba43-fa137d3cca2b-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:00.898520 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:00.898519 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a98fb70-3ec5-4999-ba43-fa137d3cca2b-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:00.899092 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:00.898530 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bxjv\" (UniqueName: \"kubernetes.io/projected/9a98fb70-3ec5-4999-ba43-fa137d3cca2b-kube-api-access-2bxjv\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:00.899092 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:00.898542 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a98fb70-3ec5-4999-ba43-fa137d3cca2b-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:01.725863 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:01.725829 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh" Nov 03 00:39:01.726085 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:01.725858 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh" event={"ID":"9a98fb70-3ec5-4999-ba43-fa137d3cca2b","Type":"ContainerDied","Data":"cc2aba5dd2568252e6eb33e1d4633552592c157eb60cd65dec11fdc0bde7aa19"} Nov 03 00:39:01.726085 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:01.725918 5727 scope.go:122] "RemoveContainer" containerID="14b2c2023385dca8b43217a96b4a857f5e4bedd4a9972ffd5879656bb34740bc" Nov 03 00:39:01.890279 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:01.890250 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh"] Nov 03 00:39:01.939959 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:01.939932 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-649f61-6dcf877cc4-9gwsh"] Nov 03 00:39:03.883294 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:03.883254 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a98fb70-3ec5-4999-ba43-fa137d3cca2b" path="/var/lib/kubelet/pods/9a98fb70-3ec5-4999-ba43-fa137d3cca2b/volumes" Nov 03 00:39:04.801950 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:04.801916 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2mxgc"] Nov 03 00:39:04.850915 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:04.850883 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2mxgc"] Nov 03 00:39:04.927080 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:04.927045 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vwlqh"] Nov 03 00:39:04.927511 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:04.927497 5727 cpu_manager.go:414] "containerMap: removing container" podUID="9a98fb70-3ec5-4999-ba43-fa137d3cca2b" containerName="maas-api" Nov 03 00:39:04.927560 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:04.927512 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a98fb70-3ec5-4999-ba43-fa137d3cca2b" containerName="maas-api" Nov 03 00:39:04.927643 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:04.927633 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a98fb70-3ec5-4999-ba43-fa137d3cca2b" containerName="maas-api" Nov 03 00:39:04.932567 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:04.932551 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vwlqh" Nov 03 00:39:04.984688 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:04.984663 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:39:04.984867 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:04.984686 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:39:04.984962 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:04.984734 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:39:04.985046 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:04.984834 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:39:04.987931 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:04.987906 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vwlqh"] Nov 03 00:39:05.039468 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:05.039434 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46xwt\" (UniqueName: \"kubernetes.io/projected/958dc25d-2014-4591-ba36-03001eb02e61-kube-api-access-46xwt\") pod \"maas-api-revoke-sub-3d4156f1-vwlqh\" (UID: \"958dc25d-2014-4591-ba36-03001eb02e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vwlqh" Nov 03 00:39:05.039647 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:05.039520 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/958dc25d-2014-4591-ba36-03001eb02e61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vwlqh\" (UID: \"958dc25d-2014-4591-ba36-03001eb02e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vwlqh" Nov 03 00:39:05.039647 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:05.039623 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/958dc25d-2014-4591-ba36-03001eb02e61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vwlqh\" (UID: \"958dc25d-2014-4591-ba36-03001eb02e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vwlqh" Nov 03 00:39:05.140231 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:05.140205 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46xwt\" (UniqueName: \"kubernetes.io/projected/958dc25d-2014-4591-ba36-03001eb02e61-kube-api-access-46xwt\") pod \"maas-api-revoke-sub-3d4156f1-vwlqh\" (UID: \"958dc25d-2014-4591-ba36-03001eb02e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vwlqh" Nov 03 00:39:05.140386 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:05.140315 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/958dc25d-2014-4591-ba36-03001eb02e61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vwlqh\" (UID: \"958dc25d-2014-4591-ba36-03001eb02e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vwlqh" Nov 03 00:39:05.140386 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:05.140351 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/958dc25d-2014-4591-ba36-03001eb02e61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vwlqh\" (UID: \"958dc25d-2014-4591-ba36-03001eb02e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vwlqh" Nov 03 00:39:05.141027 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:05.141001 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/958dc25d-2014-4591-ba36-03001eb02e61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vwlqh\" (UID: \"958dc25d-2014-4591-ba36-03001eb02e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vwlqh" Nov 03 00:39:05.142871 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:05.142848 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/958dc25d-2014-4591-ba36-03001eb02e61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vwlqh\" (UID: \"958dc25d-2014-4591-ba36-03001eb02e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vwlqh" Nov 03 00:39:05.200512 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:05.200485 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46xwt\" (UniqueName: \"kubernetes.io/projected/958dc25d-2014-4591-ba36-03001eb02e61-kube-api-access-46xwt\") pod \"maas-api-revoke-sub-3d4156f1-vwlqh\" (UID: \"958dc25d-2014-4591-ba36-03001eb02e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vwlqh" Nov 03 00:39:05.246667 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:05.246643 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vwlqh" Nov 03 00:39:05.620355 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:05.620328 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vwlqh"] Nov 03 00:39:05.622079 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:39:05.622053 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod958dc25d_2014_4591_ba36_03001eb02e61.slice/crio-bf08902d4911ce6cea89a88e873a6e45a1405a327a6ceddf74327dd61781b74b WatchSource:0}: Error finding container bf08902d4911ce6cea89a88e873a6e45a1405a327a6ceddf74327dd61781b74b: Status 404 returned error can't find the container with id bf08902d4911ce6cea89a88e873a6e45a1405a327a6ceddf74327dd61781b74b Nov 03 00:39:05.763111 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:05.763063 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vwlqh" event={"ID":"958dc25d-2014-4591-ba36-03001eb02e61","Type":"ContainerStarted","Data":"bf08902d4911ce6cea89a88e873a6e45a1405a327a6ceddf74327dd61781b74b"} Nov 03 00:39:05.883031 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:05.882941 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f3f48f3-fc54-4b65-822c-7fafd58b03a6" path="/var/lib/kubelet/pods/2f3f48f3-fc54-4b65-822c-7fafd58b03a6/volumes" Nov 03 00:39:06.771957 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:06.771921 5727 generic.go:356] "Generic (PLEG): container finished" podID="958dc25d-2014-4591-ba36-03001eb02e61" containerID="2ec203d3442c4e61a9b585bee8f02e4fb7603362543e3ba009e84b97245965ba" exitCode=0 Nov 03 00:39:06.772384 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:06.771986 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vwlqh" event={"ID":"958dc25d-2014-4591-ba36-03001eb02e61","Type":"ContainerDied","Data":"2ec203d3442c4e61a9b585bee8f02e4fb7603362543e3ba009e84b97245965ba"} Nov 03 00:39:07.921377 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:07.921338 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vwlqh" Nov 03 00:39:07.969094 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:07.969058 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/958dc25d-2014-4591-ba36-03001eb02e61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/958dc25d-2014-4591-ba36-03001eb02e61-maas-api-xks-ca\") pod \"958dc25d-2014-4591-ba36-03001eb02e61\" (UID: \"958dc25d-2014-4591-ba36-03001eb02e61\") " Nov 03 00:39:07.969289 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:07.969184 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/958dc25d-2014-4591-ba36-03001eb02e61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/958dc25d-2014-4591-ba36-03001eb02e61-maas-api-openshift-ca\") pod \"958dc25d-2014-4591-ba36-03001eb02e61\" (UID: \"958dc25d-2014-4591-ba36-03001eb02e61\") " Nov 03 00:39:07.969289 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:07.969267 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/958dc25d-2014-4591-ba36-03001eb02e61-kube-api-access-46xwt\" (UniqueName: \"kubernetes.io/projected/958dc25d-2014-4591-ba36-03001eb02e61-kube-api-access-46xwt\") pod \"958dc25d-2014-4591-ba36-03001eb02e61\" (UID: \"958dc25d-2014-4591-ba36-03001eb02e61\") " Nov 03 00:39:07.969599 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:07.969574 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/958dc25d-2014-4591-ba36-03001eb02e61-maas-api-openshift-ca" pod "958dc25d-2014-4591-ba36-03001eb02e61" (UID: "958dc25d-2014-4591-ba36-03001eb02e61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:39:07.971403 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:07.971369 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/958dc25d-2014-4591-ba36-03001eb02e61-maas-api-xks-ca" pod "958dc25d-2014-4591-ba36-03001eb02e61" (UID: "958dc25d-2014-4591-ba36-03001eb02e61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:39:07.971669 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:07.971647 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/958dc25d-2014-4591-ba36-03001eb02e61-kube-api-access-46xwt" pod "958dc25d-2014-4591-ba36-03001eb02e61" (UID: "958dc25d-2014-4591-ba36-03001eb02e61"). InnerVolumeSpecName "kube-api-access-46xwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:39:08.070926 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:08.070890 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/958dc25d-2014-4591-ba36-03001eb02e61-maas-api-xks-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:08.071072 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:08.070928 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/958dc25d-2014-4591-ba36-03001eb02e61-maas-api-openshift-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:08.071072 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:08.070945 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46xwt\" (UniqueName: \"kubernetes.io/projected/958dc25d-2014-4591-ba36-03001eb02e61-kube-api-access-46xwt\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:08.786747 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:08.786708 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vwlqh" Nov 03 00:39:08.786939 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:08.786710 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vwlqh" event={"ID":"958dc25d-2014-4591-ba36-03001eb02e61","Type":"ContainerDied","Data":"bf08902d4911ce6cea89a88e873a6e45a1405a327a6ceddf74327dd61781b74b"} Nov 03 00:39:08.786939 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:08.786842 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf08902d4911ce6cea89a88e873a6e45a1405a327a6ceddf74327dd61781b74b" Nov 03 00:39:16.046221 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:16.046183 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ppzb6"] Nov 03 00:39:16.095947 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:16.095916 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ppzb6"] Nov 03 00:39:17.881589 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:17.881556 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba3816a0-ae12-45ee-848e-c9c2d2ef994d" path="/var/lib/kubelet/pods/ba3816a0-ae12-45ee-848e-c9c2d2ef994d/volumes" Nov 03 00:39:21.572257 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:21.572215 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-kvws4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:39:21.572889 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:21.572856 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" podUID="a5514e1f-1fb4-4158-85c8-f17fe2b94eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:39:23.686327 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:23.686289 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gxmgl"] Nov 03 00:39:23.686809 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:23.686483 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gxmgl" podUID="fd31caf7-718a-4852-9aeb-6dc5b8bd05c2" containerName="manager" containerID="cri-o://638349bf9cea9a72a719d91b644c2776a656762f9e1ab6603d0ca3fac70a0903" gracePeriod=10 Nov 03 00:39:24.912254 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:24.912224 5727 generic.go:356] "Generic (PLEG): container finished" podID="fd31caf7-718a-4852-9aeb-6dc5b8bd05c2" containerID="638349bf9cea9a72a719d91b644c2776a656762f9e1ab6603d0ca3fac70a0903" exitCode=0 Nov 03 00:39:24.912556 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:24.912300 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gxmgl" event={"ID":"fd31caf7-718a-4852-9aeb-6dc5b8bd05c2","Type":"ContainerDied","Data":"638349bf9cea9a72a719d91b644c2776a656762f9e1ab6603d0ca3fac70a0903"} Nov 03 00:39:25.062705 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:25.062679 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gxmgl" Nov 03 00:39:25.066779 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:25.066760 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd31caf7-718a-4852-9aeb-6dc5b8bd05c2-kube-api-access-lbfb4\" (UniqueName: \"kubernetes.io/projected/fd31caf7-718a-4852-9aeb-6dc5b8bd05c2-kube-api-access-lbfb4\") pod \"fd31caf7-718a-4852-9aeb-6dc5b8bd05c2\" (UID: \"fd31caf7-718a-4852-9aeb-6dc5b8bd05c2\") " Nov 03 00:39:25.066857 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:25.066845 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd31caf7-718a-4852-9aeb-6dc5b8bd05c2-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd31caf7-718a-4852-9aeb-6dc5b8bd05c2-extensions-socket-volume\") pod \"fd31caf7-718a-4852-9aeb-6dc5b8bd05c2\" (UID: \"fd31caf7-718a-4852-9aeb-6dc5b8bd05c2\") " Nov 03 00:39:25.067267 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:25.067242 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd31caf7-718a-4852-9aeb-6dc5b8bd05c2-extensions-socket-volume" pod "fd31caf7-718a-4852-9aeb-6dc5b8bd05c2" (UID: "fd31caf7-718a-4852-9aeb-6dc5b8bd05c2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:39:25.068893 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:25.068871 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd31caf7-718a-4852-9aeb-6dc5b8bd05c2-kube-api-access-lbfb4" pod "fd31caf7-718a-4852-9aeb-6dc5b8bd05c2" (UID: "fd31caf7-718a-4852-9aeb-6dc5b8bd05c2"). InnerVolumeSpecName "kube-api-access-lbfb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:39:25.167903 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:25.167835 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbfb4\" (UniqueName: \"kubernetes.io/projected/fd31caf7-718a-4852-9aeb-6dc5b8bd05c2-kube-api-access-lbfb4\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:25.167903 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:25.167861 5727 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd31caf7-718a-4852-9aeb-6dc5b8bd05c2-extensions-socket-volume\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:25.919848 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:25.919815 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gxmgl" event={"ID":"fd31caf7-718a-4852-9aeb-6dc5b8bd05c2","Type":"ContainerDied","Data":"6c49c18108b4cf7e8870a2ecbd92a96a4f1f9195980a66e9155f036d50b26d57"} Nov 03 00:39:25.920267 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:25.919878 5727 scope.go:122] "RemoveContainer" containerID="638349bf9cea9a72a719d91b644c2776a656762f9e1ab6603d0ca3fac70a0903" Nov 03 00:39:25.920267 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:25.919817 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gxmgl" Nov 03 00:39:26.133621 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:26.133591 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gxmgl"] Nov 03 00:39:26.183744 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:26.183671 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gxmgl"] Nov 03 00:39:27.845945 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:39:27.845908 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:39:27.881439 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:27.881411 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd31caf7-718a-4852-9aeb-6dc5b8bd05c2" path="/var/lib/kubelet/pods/fd31caf7-718a-4852-9aeb-6dc5b8bd05c2/volumes" Nov 03 00:39:36.605913 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:36.605877 5727 scope.go:122] "RemoveContainer" containerID="de021c7877237aeb307b523f9c30a6f216e567e7d981f5226b9d2e9e7195ace6" Nov 03 00:39:36.617928 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:36.617909 5727 scope.go:122] "RemoveContainer" containerID="c98a40a4434f30b35f90ca3e6b909e2bcdac343bbfd75853e212aaa81b177158" Nov 03 00:39:36.627954 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:39:36.627937 5727 scope.go:122] "RemoveContainer" containerID="a1bd0ba593557524faa7fa6d1ba965f02b65a181e3b51f62059687f713947a1c" Nov 03 00:39:57.846788 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:39:57.846745 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:40:27.267789 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:27.267749 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-z5dtm"] Nov 03 00:40:27.268319 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:27.268209 5727 cpu_manager.go:414] "containerMap: removing container" podUID="fd31caf7-718a-4852-9aeb-6dc5b8bd05c2" containerName="manager" Nov 03 00:40:27.268319 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:27.268220 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd31caf7-718a-4852-9aeb-6dc5b8bd05c2" containerName="manager" Nov 03 00:40:27.268319 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:27.268243 5727 cpu_manager.go:414] "containerMap: removing container" podUID="958dc25d-2014-4591-ba36-03001eb02e61" containerName="revoke-keys" Nov 03 00:40:27.268319 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:27.268249 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="958dc25d-2014-4591-ba36-03001eb02e61" containerName="revoke-keys" Nov 03 00:40:27.268469 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:27.268328 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd31caf7-718a-4852-9aeb-6dc5b8bd05c2" containerName="manager" Nov 03 00:40:27.268469 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:27.268341 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="958dc25d-2014-4591-ba36-03001eb02e61" containerName="revoke-keys" Nov 03 00:40:27.271650 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:27.271633 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-z5dtm" Nov 03 00:40:27.325945 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:27.325914 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-z5dtm"] Nov 03 00:40:27.354046 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:27.354017 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zn85\" (UniqueName: \"kubernetes.io/projected/50c690b0-930f-49df-993d-036cd19836a6-kube-api-access-7zn85\") pod \"kuadrant-operator-controller-manager-678d87766f-z5dtm\" (UID: \"50c690b0-930f-49df-993d-036cd19836a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-z5dtm" Nov 03 00:40:27.354201 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:27.354073 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50c690b0-930f-49df-993d-036cd19836a6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-z5dtm\" (UID: \"50c690b0-930f-49df-993d-036cd19836a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-z5dtm" Nov 03 00:40:27.372377 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:27.372353 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6vkc2\"" Nov 03 00:40:27.455079 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:27.455050 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zn85\" (UniqueName: \"kubernetes.io/projected/50c690b0-930f-49df-993d-036cd19836a6-kube-api-access-7zn85\") pod \"kuadrant-operator-controller-manager-678d87766f-z5dtm\" (UID: \"50c690b0-930f-49df-993d-036cd19836a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-z5dtm" Nov 03 00:40:27.455229 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:27.455104 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50c690b0-930f-49df-993d-036cd19836a6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-z5dtm\" (UID: \"50c690b0-930f-49df-993d-036cd19836a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-z5dtm" Nov 03 00:40:27.455458 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:27.455433 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50c690b0-930f-49df-993d-036cd19836a6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-z5dtm\" (UID: \"50c690b0-930f-49df-993d-036cd19836a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-z5dtm" Nov 03 00:40:27.528096 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:27.528040 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zn85\" (UniqueName: \"kubernetes.io/projected/50c690b0-930f-49df-993d-036cd19836a6-kube-api-access-7zn85\") pod \"kuadrant-operator-controller-manager-678d87766f-z5dtm\" (UID: \"50c690b0-930f-49df-993d-036cd19836a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-z5dtm" Nov 03 00:40:27.585558 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:27.585537 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-z5dtm" Nov 03 00:40:27.845443 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:40:27.845407 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:40:28.060272 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:28.060248 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-z5dtm"] Nov 03 00:40:28.304920 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:28.304884 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-z5dtm" event={"ID":"50c690b0-930f-49df-993d-036cd19836a6","Type":"ContainerStarted","Data":"4b577b59ffb886e685787156784b7da3614130781e618afc465710e0d80fe87b"} Nov 03 00:40:28.305342 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:28.304950 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-z5dtm" Nov 03 00:40:28.305342 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:28.304980 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-z5dtm" event={"ID":"50c690b0-930f-49df-993d-036cd19836a6","Type":"ContainerStarted","Data":"7ca88cfb77e2d567b33b67a61e507372b1520df9ee6a2567273dbca59102a35d"} Nov 03 00:40:28.426282 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:28.426175 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-z5dtm" podStartSLOduration=1.426159301 podStartE2EDuration="1.426159301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:28.42320929 +0000 UTC m=+6714.661706979" watchObservedRunningTime="2026-11-03 00:40:28.426159301 +0000 UTC m=+6714.664656944" Nov 03 00:40:36.667997 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:36.667965 5727 scope.go:122] "RemoveContainer" containerID="fbe7dab0411f2ad44ef2718f3993574b0ae513d22a29a11918ce5b320d5155da" Nov 03 00:40:36.678071 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:36.678047 5727 scope.go:122] "RemoveContainer" containerID="bfec6122040e4db0c163d5f8963fa00c14188d4a172aeab5411984c14937512b" Nov 03 00:40:36.687345 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:36.687325 5727 scope.go:122] "RemoveContainer" containerID="4f39ec27302f6c5604ec222aea3610340b3de924dcbededbb64cd92da868060b" Nov 03 00:40:37.045871 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:37.045792 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6zk75"] Nov 03 00:40:37.095303 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:37.095274 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6zk75"] Nov 03 00:40:37.882752 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:37.882709 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1828ef7-b027-42f5-b705-6e37ff37b54c" path="/var/lib/kubelet/pods/a1828ef7-b027-42f5-b705-6e37ff37b54c/volumes" Nov 03 00:40:39.314264 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:39.314237 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-z5dtm" Nov 03 00:40:48.047572 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:48.047542 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xbnh9"] Nov 03 00:40:48.097184 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:48.097150 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xbnh9"] Nov 03 00:40:49.910154 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:49.898493 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="481ba507-2925-4706-947c-898c75c7c210" path="/var/lib/kubelet/pods/481ba507-2925-4706-947c-898c75c7c210/volumes" Nov 03 00:40:52.606527 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:52.606487 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wq2h4"] Nov 03 00:40:52.612159 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:52.612105 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wq2h4" Nov 03 00:40:52.649492 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:52.649468 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ba0ed27-8650-4dcf-b91d-b0fb284b67bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-wq2h4\" (UID: \"6ba0ed27-8650-4dcf-b91d-b0fb284b67bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wq2h4" Nov 03 00:40:52.649630 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:52.649503 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ba0ed27-8650-4dcf-b91d-b0fb284b67bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-wq2h4\" (UID: \"6ba0ed27-8650-4dcf-b91d-b0fb284b67bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wq2h4" Nov 03 00:40:52.649707 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:52.649651 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xjwh\" (UniqueName: \"kubernetes.io/projected/6ba0ed27-8650-4dcf-b91d-b0fb284b67bd-kube-api-access-7xjwh\") pod \"maas-api-revoke-sub-d576e68b-wq2h4\" (UID: \"6ba0ed27-8650-4dcf-b91d-b0fb284b67bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wq2h4" Nov 03 00:40:52.664383 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:52.664349 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:40:52.667598 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:52.667569 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wq2h4"] Nov 03 00:40:52.713862 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:52.713825 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:40:52.713862 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:52.713843 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:40:52.713862 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:52.713845 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:40:52.750704 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:52.750675 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xjwh\" (UniqueName: \"kubernetes.io/projected/6ba0ed27-8650-4dcf-b91d-b0fb284b67bd-kube-api-access-7xjwh\") pod \"maas-api-revoke-sub-d576e68b-wq2h4\" (UID: \"6ba0ed27-8650-4dcf-b91d-b0fb284b67bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wq2h4" Nov 03 00:40:52.750887 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:52.750770 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ba0ed27-8650-4dcf-b91d-b0fb284b67bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-wq2h4\" (UID: \"6ba0ed27-8650-4dcf-b91d-b0fb284b67bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wq2h4" Nov 03 00:40:52.750887 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:52.750796 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ba0ed27-8650-4dcf-b91d-b0fb284b67bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-wq2h4\" (UID: \"6ba0ed27-8650-4dcf-b91d-b0fb284b67bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wq2h4" Nov 03 00:40:52.751465 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:52.751432 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ba0ed27-8650-4dcf-b91d-b0fb284b67bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-wq2h4\" (UID: \"6ba0ed27-8650-4dcf-b91d-b0fb284b67bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wq2h4" Nov 03 00:40:52.753361 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:52.753336 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ba0ed27-8650-4dcf-b91d-b0fb284b67bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-wq2h4\" (UID: \"6ba0ed27-8650-4dcf-b91d-b0fb284b67bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wq2h4" Nov 03 00:40:52.811772 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:52.811748 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xjwh\" (UniqueName: \"kubernetes.io/projected/6ba0ed27-8650-4dcf-b91d-b0fb284b67bd-kube-api-access-7xjwh\") pod \"maas-api-revoke-sub-d576e68b-wq2h4\" (UID: \"6ba0ed27-8650-4dcf-b91d-b0fb284b67bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wq2h4" Nov 03 00:40:52.925572 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:52.925503 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wq2h4" Nov 03 00:40:53.301209 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:53.301177 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wq2h4"] Nov 03 00:40:53.302898 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:40:53.302873 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba0ed27_8650_4dcf_b91d_b0fb284b67bd.slice/crio-a8bbfa142ac8cefdcfa96dad4976ebe1636e5fce4f117a6311bc8fdf701d1b6d WatchSource:0}: Error finding container a8bbfa142ac8cefdcfa96dad4976ebe1636e5fce4f117a6311bc8fdf701d1b6d: Status 404 returned error can't find the container with id a8bbfa142ac8cefdcfa96dad4976ebe1636e5fce4f117a6311bc8fdf701d1b6d Nov 03 00:40:53.465602 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:53.465567 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wq2h4" event={"ID":"6ba0ed27-8650-4dcf-b91d-b0fb284b67bd","Type":"ContainerStarted","Data":"a8bbfa142ac8cefdcfa96dad4976ebe1636e5fce4f117a6311bc8fdf701d1b6d"} Nov 03 00:40:54.474050 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:54.474012 5727 generic.go:356] "Generic (PLEG): container finished" podID="6ba0ed27-8650-4dcf-b91d-b0fb284b67bd" containerID="9378b4e371616707e26562442064bca006000fd2496c8ec7bf207a06aa5ae40f" exitCode=0 Nov 03 00:40:54.474540 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:54.474091 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wq2h4" event={"ID":"6ba0ed27-8650-4dcf-b91d-b0fb284b67bd","Type":"ContainerDied","Data":"9378b4e371616707e26562442064bca006000fd2496c8ec7bf207a06aa5ae40f"} Nov 03 00:40:55.635261 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:55.635234 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wq2h4" Nov 03 00:40:55.678022 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:55.677988 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ba0ed27-8650-4dcf-b91d-b0fb284b67bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ba0ed27-8650-4dcf-b91d-b0fb284b67bd-maas-api-openshift-ca\") pod \"6ba0ed27-8650-4dcf-b91d-b0fb284b67bd\" (UID: \"6ba0ed27-8650-4dcf-b91d-b0fb284b67bd\") " Nov 03 00:40:55.678201 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:55.678076 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ba0ed27-8650-4dcf-b91d-b0fb284b67bd-kube-api-access-7xjwh\" (UniqueName: \"kubernetes.io/projected/6ba0ed27-8650-4dcf-b91d-b0fb284b67bd-kube-api-access-7xjwh\") pod \"6ba0ed27-8650-4dcf-b91d-b0fb284b67bd\" (UID: \"6ba0ed27-8650-4dcf-b91d-b0fb284b67bd\") " Nov 03 00:40:55.678201 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:55.678115 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ba0ed27-8650-4dcf-b91d-b0fb284b67bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ba0ed27-8650-4dcf-b91d-b0fb284b67bd-maas-api-xks-ca\") pod \"6ba0ed27-8650-4dcf-b91d-b0fb284b67bd\" (UID: \"6ba0ed27-8650-4dcf-b91d-b0fb284b67bd\") " Nov 03 00:40:55.678357 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:55.678330 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ba0ed27-8650-4dcf-b91d-b0fb284b67bd-maas-api-openshift-ca" pod "6ba0ed27-8650-4dcf-b91d-b0fb284b67bd" (UID: "6ba0ed27-8650-4dcf-b91d-b0fb284b67bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:40:55.678512 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:55.678493 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ba0ed27-8650-4dcf-b91d-b0fb284b67bd-maas-api-openshift-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:40:55.680383 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:55.680358 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba0ed27-8650-4dcf-b91d-b0fb284b67bd-kube-api-access-7xjwh" pod "6ba0ed27-8650-4dcf-b91d-b0fb284b67bd" (UID: "6ba0ed27-8650-4dcf-b91d-b0fb284b67bd"). InnerVolumeSpecName "kube-api-access-7xjwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:40:55.680486 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:55.680390 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ba0ed27-8650-4dcf-b91d-b0fb284b67bd-maas-api-xks-ca" pod "6ba0ed27-8650-4dcf-b91d-b0fb284b67bd" (UID: "6ba0ed27-8650-4dcf-b91d-b0fb284b67bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:40:55.779266 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:55.779182 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xjwh\" (UniqueName: \"kubernetes.io/projected/6ba0ed27-8650-4dcf-b91d-b0fb284b67bd-kube-api-access-7xjwh\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:40:55.779266 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:55.779224 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ba0ed27-8650-4dcf-b91d-b0fb284b67bd-maas-api-xks-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:40:56.489173 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:56.489120 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wq2h4" Nov 03 00:40:56.489173 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:56.489142 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wq2h4" event={"ID":"6ba0ed27-8650-4dcf-b91d-b0fb284b67bd","Type":"ContainerDied","Data":"a8bbfa142ac8cefdcfa96dad4976ebe1636e5fce4f117a6311bc8fdf701d1b6d"} Nov 03 00:40:56.489404 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:40:56.489200 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8bbfa142ac8cefdcfa96dad4976ebe1636e5fce4f117a6311bc8fdf701d1b6d" Nov 03 00:40:57.844928 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:40:57.844893 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:41:12.849322 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:12.849275 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-gcxrh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:41:12.849737 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:12.849346 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gcxrh" podUID="c4c2f4f5-743d-4e9d-ae63-1769c521b4ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:41:13.789408 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:13.789368 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bvws4"] Nov 03 00:41:13.790088 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:13.790070 5727 cpu_manager.go:414] "containerMap: removing container" podUID="6ba0ed27-8650-4dcf-b91d-b0fb284b67bd" containerName="revoke-keys" Nov 03 00:41:13.790184 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:13.790093 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ba0ed27-8650-4dcf-b91d-b0fb284b67bd" containerName="revoke-keys" Nov 03 00:41:13.790272 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:13.790260 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ba0ed27-8650-4dcf-b91d-b0fb284b67bd" containerName="revoke-keys" Nov 03 00:41:13.793950 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:13.793933 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bvws4" Nov 03 00:41:13.848763 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:13.848725 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bvws4"] Nov 03 00:41:13.894762 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:13.894732 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:41:13.895177 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:13.894776 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:41:13.895177 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:13.894811 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:41:13.895398 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:13.895386 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:41:13.940562 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:13.940529 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8bf8352-2cad-4cd1-b3b9-9776448f0f5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-bvws4\" (UID: \"a8bf8352-2cad-4cd1-b3b9-9776448f0f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bvws4" Nov 03 00:41:13.940741 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:13.940634 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2btpn\" (UniqueName: \"kubernetes.io/projected/a8bf8352-2cad-4cd1-b3b9-9776448f0f5e-kube-api-access-2btpn\") pod \"maas-api-revoke-sub-d2bc93bc-bvws4\" (UID: \"a8bf8352-2cad-4cd1-b3b9-9776448f0f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bvws4" Nov 03 00:41:13.940811 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:13.940733 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8bf8352-2cad-4cd1-b3b9-9776448f0f5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-bvws4\" (UID: \"a8bf8352-2cad-4cd1-b3b9-9776448f0f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bvws4" Nov 03 00:41:14.042094 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:14.042021 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8bf8352-2cad-4cd1-b3b9-9776448f0f5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-bvws4\" (UID: \"a8bf8352-2cad-4cd1-b3b9-9776448f0f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bvws4" Nov 03 00:41:14.042094 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:14.042068 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2btpn\" (UniqueName: \"kubernetes.io/projected/a8bf8352-2cad-4cd1-b3b9-9776448f0f5e-kube-api-access-2btpn\") pod \"maas-api-revoke-sub-d2bc93bc-bvws4\" (UID: \"a8bf8352-2cad-4cd1-b3b9-9776448f0f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bvws4" Nov 03 00:41:14.042361 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:14.042116 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8bf8352-2cad-4cd1-b3b9-9776448f0f5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-bvws4\" (UID: \"a8bf8352-2cad-4cd1-b3b9-9776448f0f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bvws4" Nov 03 00:41:14.042677 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:14.042655 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8bf8352-2cad-4cd1-b3b9-9776448f0f5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-bvws4\" (UID: \"a8bf8352-2cad-4cd1-b3b9-9776448f0f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bvws4" Nov 03 00:41:14.044686 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:14.044664 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8bf8352-2cad-4cd1-b3b9-9776448f0f5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-bvws4\" (UID: \"a8bf8352-2cad-4cd1-b3b9-9776448f0f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bvws4" Nov 03 00:41:14.103190 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:14.103167 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2btpn\" (UniqueName: \"kubernetes.io/projected/a8bf8352-2cad-4cd1-b3b9-9776448f0f5e-kube-api-access-2btpn\") pod \"maas-api-revoke-sub-d2bc93bc-bvws4\" (UID: \"a8bf8352-2cad-4cd1-b3b9-9776448f0f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bvws4" Nov 03 00:41:14.107929 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:14.107911 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bvws4" Nov 03 00:41:14.481421 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:14.481392 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bvws4"] Nov 03 00:41:14.483483 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:41:14.483447 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8bf8352_2cad_4cd1_b3b9_9776448f0f5e.slice/crio-cd098f9fc7b37f89fc4872511f22c4d3ab3049f4e5653e509d466d14efee9d84 WatchSource:0}: Error finding container cd098f9fc7b37f89fc4872511f22c4d3ab3049f4e5653e509d466d14efee9d84: Status 404 returned error can't find the container with id cd098f9fc7b37f89fc4872511f22c4d3ab3049f4e5653e509d466d14efee9d84 Nov 03 00:41:14.633463 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:14.633420 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bvws4" event={"ID":"a8bf8352-2cad-4cd1-b3b9-9776448f0f5e","Type":"ContainerStarted","Data":"cd098f9fc7b37f89fc4872511f22c4d3ab3049f4e5653e509d466d14efee9d84"} Nov 03 00:41:15.641714 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:15.641625 5727 generic.go:356] "Generic (PLEG): container finished" podID="a8bf8352-2cad-4cd1-b3b9-9776448f0f5e" containerID="3ecb45c2ddfc837c6587f57ceeb60d041ea49b9300205c53254611ba5c925712" exitCode=0 Nov 03 00:41:15.642083 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:15.641717 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bvws4" event={"ID":"a8bf8352-2cad-4cd1-b3b9-9776448f0f5e","Type":"ContainerDied","Data":"3ecb45c2ddfc837c6587f57ceeb60d041ea49b9300205c53254611ba5c925712"} Nov 03 00:41:16.794557 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:16.794529 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bvws4" Nov 03 00:41:16.975336 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:16.975247 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8bf8352-2cad-4cd1-b3b9-9776448f0f5e-kube-api-access-2btpn\" (UniqueName: \"kubernetes.io/projected/a8bf8352-2cad-4cd1-b3b9-9776448f0f5e-kube-api-access-2btpn\") pod \"a8bf8352-2cad-4cd1-b3b9-9776448f0f5e\" (UID: \"a8bf8352-2cad-4cd1-b3b9-9776448f0f5e\") " Nov 03 00:41:16.975336 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:16.975331 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8bf8352-2cad-4cd1-b3b9-9776448f0f5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8bf8352-2cad-4cd1-b3b9-9776448f0f5e-maas-api-xks-ca\") pod \"a8bf8352-2cad-4cd1-b3b9-9776448f0f5e\" (UID: \"a8bf8352-2cad-4cd1-b3b9-9776448f0f5e\") " Nov 03 00:41:16.975602 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:16.975425 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8bf8352-2cad-4cd1-b3b9-9776448f0f5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8bf8352-2cad-4cd1-b3b9-9776448f0f5e-maas-api-openshift-ca\") pod \"a8bf8352-2cad-4cd1-b3b9-9776448f0f5e\" (UID: \"a8bf8352-2cad-4cd1-b3b9-9776448f0f5e\") " Nov 03 00:41:16.975788 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:16.975754 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8bf8352-2cad-4cd1-b3b9-9776448f0f5e-maas-api-openshift-ca" pod "a8bf8352-2cad-4cd1-b3b9-9776448f0f5e" (UID: "a8bf8352-2cad-4cd1-b3b9-9776448f0f5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:41:16.975882 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:16.975843 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8bf8352-2cad-4cd1-b3b9-9776448f0f5e-maas-api-openshift-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:16.977659 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:16.977625 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8bf8352-2cad-4cd1-b3b9-9776448f0f5e-maas-api-xks-ca" pod "a8bf8352-2cad-4cd1-b3b9-9776448f0f5e" (UID: "a8bf8352-2cad-4cd1-b3b9-9776448f0f5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:41:16.977809 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:16.977693 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8bf8352-2cad-4cd1-b3b9-9776448f0f5e-kube-api-access-2btpn" pod "a8bf8352-2cad-4cd1-b3b9-9776448f0f5e" (UID: "a8bf8352-2cad-4cd1-b3b9-9776448f0f5e"). InnerVolumeSpecName "kube-api-access-2btpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:41:17.076606 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:17.076563 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2btpn\" (UniqueName: \"kubernetes.io/projected/a8bf8352-2cad-4cd1-b3b9-9776448f0f5e-kube-api-access-2btpn\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:17.076606 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:17.076606 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8bf8352-2cad-4cd1-b3b9-9776448f0f5e-maas-api-xks-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:17.656404 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:17.656372 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bvws4" Nov 03 00:41:17.656598 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:17.656414 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bvws4" event={"ID":"a8bf8352-2cad-4cd1-b3b9-9776448f0f5e","Type":"ContainerDied","Data":"cd098f9fc7b37f89fc4872511f22c4d3ab3049f4e5653e509d466d14efee9d84"} Nov 03 00:41:17.656598 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:17.656470 5727 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd098f9fc7b37f89fc4872511f22c4d3ab3049f4e5653e509d466d14efee9d84" Nov 03 00:41:26.845195 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:26.845155 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-94b5f474-d2jvt"] Nov 03 00:41:26.845722 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:26.845627 5727 cpu_manager.go:414] "containerMap: removing container" podUID="a8bf8352-2cad-4cd1-b3b9-9776448f0f5e" containerName="revoke-keys" Nov 03 00:41:26.845722 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:26.845637 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8bf8352-2cad-4cd1-b3b9-9776448f0f5e" containerName="revoke-keys" Nov 03 00:41:26.845722 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:26.845713 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8bf8352-2cad-4cd1-b3b9-9776448f0f5e" containerName="revoke-keys" Nov 03 00:41:26.849179 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:26.849154 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-94b5f474-d2jvt" Nov 03 00:41:26.868175 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:26.868119 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec588183-be39-4d98-b4bc-85d74ae54e26-metrics-certs\") pod \"maas-api-94b5f474-d2jvt\" (UID: \"ec588183-be39-4d98-b4bc-85d74ae54e26\") " pod="odh-ai-gateway-infra/maas-api-94b5f474-d2jvt" Nov 03 00:41:26.868293 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:26.868214 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec588183-be39-4d98-b4bc-85d74ae54e26-openshift-service-ca\") pod \"maas-api-94b5f474-d2jvt\" (UID: \"ec588183-be39-4d98-b4bc-85d74ae54e26\") " pod="odh-ai-gateway-infra/maas-api-94b5f474-d2jvt" Nov 03 00:41:26.868293 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:26.868243 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec588183-be39-4d98-b4bc-85d74ae54e26-maas-api-tls\") pod \"maas-api-94b5f474-d2jvt\" (UID: \"ec588183-be39-4d98-b4bc-85d74ae54e26\") " pod="odh-ai-gateway-infra/maas-api-94b5f474-d2jvt" Nov 03 00:41:26.868441 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:26.868420 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wvsh\" (UniqueName: \"kubernetes.io/projected/ec588183-be39-4d98-b4bc-85d74ae54e26-kube-api-access-9wvsh\") pod \"maas-api-94b5f474-d2jvt\" (UID: \"ec588183-be39-4d98-b4bc-85d74ae54e26\") " pod="odh-ai-gateway-infra/maas-api-94b5f474-d2jvt" Nov 03 00:41:26.950812 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:26.950784 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:41:26.950990 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:26.950858 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 00:41:26.951043 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:26.951021 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wm4gz\"" Nov 03 00:41:26.951083 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:26.951049 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:41:26.951083 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:26.951065 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 00:41:26.954247 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:26.954224 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-94b5f474-d2jvt"] Nov 03 00:41:26.969633 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:26.969602 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec588183-be39-4d98-b4bc-85d74ae54e26-metrics-certs\") pod \"maas-api-94b5f474-d2jvt\" (UID: \"ec588183-be39-4d98-b4bc-85d74ae54e26\") " pod="odh-ai-gateway-infra/maas-api-94b5f474-d2jvt" Nov 03 00:41:26.969781 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:26.969646 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec588183-be39-4d98-b4bc-85d74ae54e26-openshift-service-ca\") pod \"maas-api-94b5f474-d2jvt\" (UID: \"ec588183-be39-4d98-b4bc-85d74ae54e26\") " pod="odh-ai-gateway-infra/maas-api-94b5f474-d2jvt" Nov 03 00:41:26.969781 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:26.969672 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec588183-be39-4d98-b4bc-85d74ae54e26-maas-api-tls\") pod \"maas-api-94b5f474-d2jvt\" (UID: \"ec588183-be39-4d98-b4bc-85d74ae54e26\") " pod="odh-ai-gateway-infra/maas-api-94b5f474-d2jvt" Nov 03 00:41:26.969898 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:26.969806 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wvsh\" (UniqueName: \"kubernetes.io/projected/ec588183-be39-4d98-b4bc-85d74ae54e26-kube-api-access-9wvsh\") pod \"maas-api-94b5f474-d2jvt\" (UID: \"ec588183-be39-4d98-b4bc-85d74ae54e26\") " pod="odh-ai-gateway-infra/maas-api-94b5f474-d2jvt" Nov 03 00:41:26.970617 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:26.970523 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec588183-be39-4d98-b4bc-85d74ae54e26-openshift-service-ca\") pod \"maas-api-94b5f474-d2jvt\" (UID: \"ec588183-be39-4d98-b4bc-85d74ae54e26\") " pod="odh-ai-gateway-infra/maas-api-94b5f474-d2jvt" Nov 03 00:41:26.972454 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:26.972428 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec588183-be39-4d98-b4bc-85d74ae54e26-maas-api-tls\") pod \"maas-api-94b5f474-d2jvt\" (UID: \"ec588183-be39-4d98-b4bc-85d74ae54e26\") " pod="odh-ai-gateway-infra/maas-api-94b5f474-d2jvt" Nov 03 00:41:26.972623 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:26.972602 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec588183-be39-4d98-b4bc-85d74ae54e26-metrics-certs\") pod \"maas-api-94b5f474-d2jvt\" (UID: \"ec588183-be39-4d98-b4bc-85d74ae54e26\") " pod="odh-ai-gateway-infra/maas-api-94b5f474-d2jvt" Nov 03 00:41:27.030529 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:27.030502 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wvsh\" (UniqueName: \"kubernetes.io/projected/ec588183-be39-4d98-b4bc-85d74ae54e26-kube-api-access-9wvsh\") pod \"maas-api-94b5f474-d2jvt\" (UID: \"ec588183-be39-4d98-b4bc-85d74ae54e26\") " pod="odh-ai-gateway-infra/maas-api-94b5f474-d2jvt" Nov 03 00:41:27.164236 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:27.164123 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-94b5f474-d2jvt" Nov 03 00:41:27.543603 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:27.543578 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-94b5f474-d2jvt"] Nov 03 00:41:27.545520 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:41:27.545492 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec588183_be39_4d98_b4bc_85d74ae54e26.slice/crio-e634f581ba4e37d99e8c48e36990d7abe7f8fd945cdcef1b847d70e9aae76bad WatchSource:0}: Error finding container e634f581ba4e37d99e8c48e36990d7abe7f8fd945cdcef1b847d70e9aae76bad: Status 404 returned error can't find the container with id e634f581ba4e37d99e8c48e36990d7abe7f8fd945cdcef1b847d70e9aae76bad Nov 03 00:41:27.729357 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:27.729323 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-94b5f474-d2jvt" event={"ID":"ec588183-be39-4d98-b4bc-85d74ae54e26","Type":"ContainerStarted","Data":"e634f581ba4e37d99e8c48e36990d7abe7f8fd945cdcef1b847d70e9aae76bad"} Nov 03 00:41:27.845388 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:41:27.845353 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:41:28.737380 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:28.737291 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-94b5f474-d2jvt" event={"ID":"ec588183-be39-4d98-b4bc-85d74ae54e26","Type":"ContainerStarted","Data":"01b7aea448f48a69c3cec6b783cf12a47b2b3632349727b7c2fa69c27ad8bb09"} Nov 03 00:41:28.737538 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:28.737500 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-94b5f474-d2jvt" Nov 03 00:41:28.855246 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:28.855192 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-94b5f474-d2jvt" podStartSLOduration=1.964780899 podStartE2EDuration="2.855177638s" totalImagesPullingTime="890.396739ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:28.853960462 +0000 UTC m=+6775.092458162" watchObservedRunningTime="2026-11-03 00:41:28.855177638 +0000 UTC m=+6775.093675300" Nov 03 00:41:34.749119 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:34.749091 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-94b5f474-d2jvt" Nov 03 00:41:34.855909 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:34.855881 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-94b5f474-d2jvt"] Nov 03 00:41:34.856078 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:34.856059 5727 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-94b5f474-d2jvt" podUID="ec588183-be39-4d98-b4bc-85d74ae54e26" containerName="maas-api" containerID="cri-o://01b7aea448f48a69c3cec6b783cf12a47b2b3632349727b7c2fa69c27ad8bb09" gracePeriod=30 Nov 03 00:41:34.869701 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:34.869669 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-69c7f9d898-448gl"] Nov 03 00:41:34.874338 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:34.874305 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-69c7f9d898-448gl" Nov 03 00:41:34.949858 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:34.949819 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dffa001d-3077-4159-95f3-47be4e4fbdd5-openshift-service-ca\") pod \"maas-api-69c7f9d898-448gl\" (UID: \"dffa001d-3077-4159-95f3-47be4e4fbdd5\") " pod="odh-ai-gateway-infra/maas-api-69c7f9d898-448gl" Nov 03 00:41:34.950033 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:34.949868 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2qxh\" (UniqueName: \"kubernetes.io/projected/dffa001d-3077-4159-95f3-47be4e4fbdd5-kube-api-access-j2qxh\") pod \"maas-api-69c7f9d898-448gl\" (UID: \"dffa001d-3077-4159-95f3-47be4e4fbdd5\") " pod="odh-ai-gateway-infra/maas-api-69c7f9d898-448gl" Nov 03 00:41:34.950033 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:34.949941 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dffa001d-3077-4159-95f3-47be4e4fbdd5-maas-api-tls\") pod \"maas-api-69c7f9d898-448gl\" (UID: \"dffa001d-3077-4159-95f3-47be4e4fbdd5\") " pod="odh-ai-gateway-infra/maas-api-69c7f9d898-448gl" Nov 03 00:41:34.950033 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:34.950005 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dffa001d-3077-4159-95f3-47be4e4fbdd5-metrics-certs\") pod \"maas-api-69c7f9d898-448gl\" (UID: \"dffa001d-3077-4159-95f3-47be4e4fbdd5\") " pod="odh-ai-gateway-infra/maas-api-69c7f9d898-448gl" Nov 03 00:41:34.967757 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:34.967722 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-69c7f9d898-448gl"] Nov 03 00:41:35.051883 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.051836 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dffa001d-3077-4159-95f3-47be4e4fbdd5-metrics-certs\") pod \"maas-api-69c7f9d898-448gl\" (UID: \"dffa001d-3077-4159-95f3-47be4e4fbdd5\") " pod="odh-ai-gateway-infra/maas-api-69c7f9d898-448gl" Nov 03 00:41:35.052081 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.052020 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dffa001d-3077-4159-95f3-47be4e4fbdd5-openshift-service-ca\") pod \"maas-api-69c7f9d898-448gl\" (UID: \"dffa001d-3077-4159-95f3-47be4e4fbdd5\") " pod="odh-ai-gateway-infra/maas-api-69c7f9d898-448gl" Nov 03 00:41:35.052081 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.052064 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2qxh\" (UniqueName: \"kubernetes.io/projected/dffa001d-3077-4159-95f3-47be4e4fbdd5-kube-api-access-j2qxh\") pod \"maas-api-69c7f9d898-448gl\" (UID: \"dffa001d-3077-4159-95f3-47be4e4fbdd5\") " pod="odh-ai-gateway-infra/maas-api-69c7f9d898-448gl" Nov 03 00:41:35.052234 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.052094 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dffa001d-3077-4159-95f3-47be4e4fbdd5-maas-api-tls\") pod \"maas-api-69c7f9d898-448gl\" (UID: \"dffa001d-3077-4159-95f3-47be4e4fbdd5\") " pod="odh-ai-gateway-infra/maas-api-69c7f9d898-448gl" Nov 03 00:41:35.052853 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.052789 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dffa001d-3077-4159-95f3-47be4e4fbdd5-openshift-service-ca\") pod \"maas-api-69c7f9d898-448gl\" (UID: \"dffa001d-3077-4159-95f3-47be4e4fbdd5\") " pod="odh-ai-gateway-infra/maas-api-69c7f9d898-448gl" Nov 03 00:41:35.054638 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.054538 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dffa001d-3077-4159-95f3-47be4e4fbdd5-maas-api-tls\") pod \"maas-api-69c7f9d898-448gl\" (UID: \"dffa001d-3077-4159-95f3-47be4e4fbdd5\") " pod="odh-ai-gateway-infra/maas-api-69c7f9d898-448gl" Nov 03 00:41:35.055511 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.055347 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dffa001d-3077-4159-95f3-47be4e4fbdd5-metrics-certs\") pod \"maas-api-69c7f9d898-448gl\" (UID: \"dffa001d-3077-4159-95f3-47be4e4fbdd5\") " pod="odh-ai-gateway-infra/maas-api-69c7f9d898-448gl" Nov 03 00:41:35.112745 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.112710 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2qxh\" (UniqueName: \"kubernetes.io/projected/dffa001d-3077-4159-95f3-47be4e4fbdd5-kube-api-access-j2qxh\") pod \"maas-api-69c7f9d898-448gl\" (UID: \"dffa001d-3077-4159-95f3-47be4e4fbdd5\") " pod="odh-ai-gateway-infra/maas-api-69c7f9d898-448gl" Nov 03 00:41:35.149746 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.149722 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-94b5f474-d2jvt" Nov 03 00:41:35.244706 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.244678 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-69c7f9d898-448gl" Nov 03 00:41:35.253425 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.253371 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec588183-be39-4d98-b4bc-85d74ae54e26-kube-api-access-9wvsh\" (UniqueName: \"kubernetes.io/projected/ec588183-be39-4d98-b4bc-85d74ae54e26-kube-api-access-9wvsh\") pod \"ec588183-be39-4d98-b4bc-85d74ae54e26\" (UID: \"ec588183-be39-4d98-b4bc-85d74ae54e26\") " Nov 03 00:41:35.253553 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.253497 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec588183-be39-4d98-b4bc-85d74ae54e26-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec588183-be39-4d98-b4bc-85d74ae54e26-openshift-service-ca\") pod \"ec588183-be39-4d98-b4bc-85d74ae54e26\" (UID: \"ec588183-be39-4d98-b4bc-85d74ae54e26\") " Nov 03 00:41:35.253624 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.253576 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec588183-be39-4d98-b4bc-85d74ae54e26-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec588183-be39-4d98-b4bc-85d74ae54e26-metrics-certs\") pod \"ec588183-be39-4d98-b4bc-85d74ae54e26\" (UID: \"ec588183-be39-4d98-b4bc-85d74ae54e26\") " Nov 03 00:41:35.253685 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.253667 5727 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec588183-be39-4d98-b4bc-85d74ae54e26-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec588183-be39-4d98-b4bc-85d74ae54e26-maas-api-tls\") pod \"ec588183-be39-4d98-b4bc-85d74ae54e26\" (UID: \"ec588183-be39-4d98-b4bc-85d74ae54e26\") " Nov 03 00:41:35.253920 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.253894 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec588183-be39-4d98-b4bc-85d74ae54e26-openshift-service-ca" pod "ec588183-be39-4d98-b4bc-85d74ae54e26" (UID: "ec588183-be39-4d98-b4bc-85d74ae54e26"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:41:35.254035 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.253999 5727 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec588183-be39-4d98-b4bc-85d74ae54e26-openshift-service-ca\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:35.255754 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.255723 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec588183-be39-4d98-b4bc-85d74ae54e26-metrics-certs" pod "ec588183-be39-4d98-b4bc-85d74ae54e26" (UID: "ec588183-be39-4d98-b4bc-85d74ae54e26"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:41:35.255754 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.255729 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec588183-be39-4d98-b4bc-85d74ae54e26-kube-api-access-9wvsh" pod "ec588183-be39-4d98-b4bc-85d74ae54e26" (UID: "ec588183-be39-4d98-b4bc-85d74ae54e26"). InnerVolumeSpecName "kube-api-access-9wvsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:41:35.255908 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.255850 5727 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec588183-be39-4d98-b4bc-85d74ae54e26-maas-api-tls" pod "ec588183-be39-4d98-b4bc-85d74ae54e26" (UID: "ec588183-be39-4d98-b4bc-85d74ae54e26"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:41:35.354862 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.354833 5727 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec588183-be39-4d98-b4bc-85d74ae54e26-metrics-certs\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:35.354862 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.354863 5727 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec588183-be39-4d98-b4bc-85d74ae54e26-maas-api-tls\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:35.355052 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.354878 5727 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wvsh\" (UniqueName: \"kubernetes.io/projected/ec588183-be39-4d98-b4bc-85d74ae54e26-kube-api-access-9wvsh\") on node \"ip-10-0-135-176.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:35.623397 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.623373 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-69c7f9d898-448gl"] Nov 03 00:41:35.624522 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:41:35.624492 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddffa001d_3077_4159_95f3_47be4e4fbdd5.slice/crio-07b2224074a90716d69fc5fd2f1ae665dc63ab2a5904aeba707577ca3f68e396 WatchSource:0}: Error finding container 07b2224074a90716d69fc5fd2f1ae665dc63ab2a5904aeba707577ca3f68e396: Status 404 returned error can't find the container with id 07b2224074a90716d69fc5fd2f1ae665dc63ab2a5904aeba707577ca3f68e396 Nov 03 00:41:35.625930 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.625910 5727 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:41:35.780520 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.780487 5727 generic.go:356] "Generic (PLEG): container finished" podID="ec588183-be39-4d98-b4bc-85d74ae54e26" containerID="01b7aea448f48a69c3cec6b783cf12a47b2b3632349727b7c2fa69c27ad8bb09" exitCode=0 Nov 03 00:41:35.781037 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.780533 5727 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-94b5f474-d2jvt" Nov 03 00:41:35.781037 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.780577 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-94b5f474-d2jvt" event={"ID":"ec588183-be39-4d98-b4bc-85d74ae54e26","Type":"ContainerDied","Data":"01b7aea448f48a69c3cec6b783cf12a47b2b3632349727b7c2fa69c27ad8bb09"} Nov 03 00:41:35.781037 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.780638 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-94b5f474-d2jvt" event={"ID":"ec588183-be39-4d98-b4bc-85d74ae54e26","Type":"ContainerDied","Data":"e634f581ba4e37d99e8c48e36990d7abe7f8fd945cdcef1b847d70e9aae76bad"} Nov 03 00:41:35.781037 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.780679 5727 scope.go:122] "RemoveContainer" containerID="01b7aea448f48a69c3cec6b783cf12a47b2b3632349727b7c2fa69c27ad8bb09" Nov 03 00:41:35.782113 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.782092 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-69c7f9d898-448gl" event={"ID":"dffa001d-3077-4159-95f3-47be4e4fbdd5","Type":"ContainerStarted","Data":"07b2224074a90716d69fc5fd2f1ae665dc63ab2a5904aeba707577ca3f68e396"} Nov 03 00:41:35.790797 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.790782 5727 scope.go:122] "RemoveContainer" containerID="01b7aea448f48a69c3cec6b783cf12a47b2b3632349727b7c2fa69c27ad8bb09" Nov 03 00:41:35.791058 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:41:35.791037 5727 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01b7aea448f48a69c3cec6b783cf12a47b2b3632349727b7c2fa69c27ad8bb09\": container with ID starting with 01b7aea448f48a69c3cec6b783cf12a47b2b3632349727b7c2fa69c27ad8bb09 not found: ID does not exist" containerID="01b7aea448f48a69c3cec6b783cf12a47b2b3632349727b7c2fa69c27ad8bb09" Nov 03 00:41:35.791107 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.791068 5727 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01b7aea448f48a69c3cec6b783cf12a47b2b3632349727b7c2fa69c27ad8bb09"} err="failed to get container status \"01b7aea448f48a69c3cec6b783cf12a47b2b3632349727b7c2fa69c27ad8bb09\": rpc error: code = NotFound desc = could not find container \"01b7aea448f48a69c3cec6b783cf12a47b2b3632349727b7c2fa69c27ad8bb09\": container with ID starting with 01b7aea448f48a69c3cec6b783cf12a47b2b3632349727b7c2fa69c27ad8bb09 not found: ID does not exist" Nov 03 00:41:35.944198 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.944171 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-94b5f474-d2jvt"] Nov 03 00:41:35.950282 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:35.950258 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-94b5f474-d2jvt"] Nov 03 00:41:36.720513 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:36.720487 5727 scope.go:122] "RemoveContainer" containerID="0c6f547db2badbdc0e996c435be14a5adf2047428b8bd800847cebfad1e88e5d" Nov 03 00:41:36.730016 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:36.729993 5727 scope.go:122] "RemoveContainer" containerID="39cde42db47ddc51dce3739e7e0bcacd361e241fc89ac5d5e2b3fc0e7478e0fd" Nov 03 00:41:36.789659 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:36.789627 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-69c7f9d898-448gl" event={"ID":"dffa001d-3077-4159-95f3-47be4e4fbdd5","Type":"ContainerStarted","Data":"e71fa746f8cd514fe29db1c5b8987b9a59672f862d7739b0cd130c902e5f77cc"} Nov 03 00:41:36.790040 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:36.789690 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-69c7f9d898-448gl" Nov 03 00:41:36.906407 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:36.906350 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-69c7f9d898-448gl" podStartSLOduration=2.060474408 podStartE2EDuration="2.906337015s" totalImagesPullingTime="845.862607ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:36.90594259 +0000 UTC m=+6783.144440271" watchObservedRunningTime="2026-11-03 00:41:36.906337015 +0000 UTC m=+6783.144834677" Nov 03 00:41:37.883961 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:37.883923 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec588183-be39-4d98-b4bc-85d74ae54e26" path="/var/lib/kubelet/pods/ec588183-be39-4d98-b4bc-85d74ae54e26/volumes" Nov 03 00:41:42.805743 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:41:42.805711 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-69c7f9d898-448gl" Nov 03 00:41:57.846042 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:41:57.846000 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:42:27.845966 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:42:27.845924 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:42:36.762710 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:42:36.762675 5727 scope.go:122] "RemoveContainer" containerID="1bf5dc6c586169700708b228ab05e5cc235d4c02f9af0dc61d61669ef80e45f2" Nov 03 00:42:36.774535 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:42:36.774512 5727 scope.go:122] "RemoveContainer" containerID="d74db0d33b70357fcb9565c58b8652fc8cd0f8abf6bbafa142b014c62b63902b" Nov 03 00:42:57.848496 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:42:57.848452 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:43:27.844734 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:43:27.844693 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:43:35.325400 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:43:35.325372 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 03 00:43:35.340653 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:43:35.340630 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 03 00:43:35.425899 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:43:35.425871 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 03 00:43:35.441814 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:43:35.441787 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 03 00:43:57.845502 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:43:57.845464 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:44:09.047258 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:44:09.047217 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vwlqh"] Nov 03 00:44:09.096727 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:44:09.096699 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vwlqh"] Nov 03 00:44:09.882300 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:44:09.882260 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="958dc25d-2014-4591-ba36-03001eb02e61" path="/var/lib/kubelet/pods/958dc25d-2014-4591-ba36-03001eb02e61/volumes" Nov 03 00:44:27.845559 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:44:27.845521 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:44:36.571312 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:44:36.571262 5727 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-kvws4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:44:36.572460 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:44:36.571947 5727 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kvws4" podUID="a5514e1f-1fb4-4158-85c8-f17fe2b94eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:44:57.845303 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:44:57.845258 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:45:27.846165 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:45:27.846109 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:45:36.834071 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:45:36.834040 5727 scope.go:122] "RemoveContainer" containerID="2ec203d3442c4e61a9b585bee8f02e4fb7603362543e3ba009e84b97245965ba" Nov 03 00:45:56.056576 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:45:56.056488 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wq2h4"] Nov 03 00:45:56.105871 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:45:56.105843 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wq2h4"] Nov 03 00:45:57.849000 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:45:57.848953 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:45:57.883408 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:45:57.883375 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ba0ed27-8650-4dcf-b91d-b0fb284b67bd" path="/var/lib/kubelet/pods/6ba0ed27-8650-4dcf-b91d-b0fb284b67bd/volumes" Nov 03 00:46:17.047408 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:46:17.047369 5727 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bvws4"] Nov 03 00:46:17.097110 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:46:17.097072 5727 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bvws4"] Nov 03 00:46:17.889035 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:46:17.888996 5727 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8bf8352-2cad-4cd1-b3b9-9776448f0f5e" path="/var/lib/kubelet/pods/a8bf8352-2cad-4cd1-b3b9-9776448f0f5e/volumes" Nov 03 00:46:27.845064 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:46:27.845024 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:46:57.845564 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:46:57.845530 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:47:27.845467 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:47:27.845425 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:47:36.867871 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:47:36.867835 5727 scope.go:122] "RemoveContainer" containerID="3ecb45c2ddfc837c6587f57ceeb60d041ea49b9300205c53254611ba5c925712" Nov 03 00:47:36.877899 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:47:36.877875 5727 scope.go:122] "RemoveContainer" containerID="9378b4e371616707e26562442064bca006000fd2496c8ec7bf207a06aa5ae40f" Nov 03 00:47:47.637543 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:47:47.637513 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-57f5d89f7c-zkm62_cf7f72a3-5ca3-4f5e-a8e2-c88f1485806a/authorino/0.log" Nov 03 00:47:57.846541 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:47:57.846471 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:47:59.336328 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:47:59.336297 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7fcfb78cb5-m2vnt_b2101b43-6e2c-42d8-8d60-d44faa6433f1/manager/0.log" Nov 03 00:47:59.692074 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:47:59.691999 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qprz9_b7266a2f-427b-47f5-9528-e8c111ae8d04/manager/0.log" Nov 03 00:47:59.917338 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:47:59.917309 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-knpxk_f601f493-313e-40b5-9e89-0c492a35255a/manager/0.log" Nov 03 00:48:01.184716 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:01.184691 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-cnzpg_b86548a3-cf54-41a9-8c65-f543ae9daa83/server/0.log" Nov 03 00:48:01.684391 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:01.684362 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-cb6tl_54f62e96-134f-44b3-839e-a02edd85ab08/proxy/0.log" Nov 03 00:48:02.697351 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:02.697322 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-69c7f9d898-448gl_dffa001d-3077-4159-95f3-47be4e4fbdd5/maas-api/0.log" Nov 03 00:48:08.697676 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:08.697641 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-57f5d89f7c-zkm62_cf7f72a3-5ca3-4f5e-a8e2-c88f1485806a/authorino/0.log" Nov 03 00:48:09.070910 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:09.070875 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-ptd66_de8616c3-ba9c-4b99-813e-ef4c190fc4c7/manager/0.log" Nov 03 00:48:09.305693 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:09.305660 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-cp9j8_e5198e9e-a261-4491-93da-4fad7b359312/manager/0.log" Nov 03 00:48:09.514900 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:09.514821 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jr9qg_1394c83b-9059-4e3e-928f-bde172669b09/kuadrant-console-plugin/0.log" Nov 03 00:48:09.894216 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:09.894184 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-z5dtm_50c690b0-930f-49df-993d-036cd19836a6/manager/0.log" Nov 03 00:48:10.436434 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:10.436402 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-x6h9v_349fa078-8c82-4c07-a596-900fbb745d7f/manager/1.log" Nov 03 00:48:11.101966 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:11.101940 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-x6h9v_349fa078-8c82-4c07-a596-900fbb745d7f/manager/0.log" Nov 03 00:48:11.675720 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:11.675680 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr_856fc499-f20f-41cd-8baf-ed58d33c45c5/istio-proxy/0.log" Nov 03 00:48:13.880559 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:13.880526 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-gcxrh_c4c2f4f5-743d-4e9d-ae63-1769c521b4ad/istio-proxy/1.log" Nov 03 00:48:14.683672 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:14.683644 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-kvws4_a5514e1f-1fb4-4158-85c8-f17fe2b94eb8/istio-proxy/0.log" Nov 03 00:48:15.227546 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:15.227516 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-vgtv2_3137ed7b-3a02-433a-b95d-7cd75469aac5/istio-proxy/0.log" Nov 03 00:48:15.450503 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:15.450479 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2_59f85494-1949-4969-8e31-c6a4b155f4ae/istio-proxy/0.log" Nov 03 00:48:15.658767 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:15.658739 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-75bc6697f4-778p5_36f33895-59f7-4533-9478-fcd8299a8b4f/payload-pre-processing/0.log" Nov 03 00:48:15.879252 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:15.879216 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc7f58bb9-9dh6m_f6ba592a-8ce9-44bd-9ade-8dfd408bad2a/payload-processing/0.log" Nov 03 00:48:16.101051 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:16.101021 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-566bb4455-kq2j4_1dccc35b-6206-47f3-b901-bafc5f17b8ad/router/0.log" Nov 03 00:48:16.777455 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:16.777426 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-gcxrh_c4c2f4f5-743d-4e9d-ae63-1769c521b4ad/istio-proxy/0.log" Nov 03 00:48:17.334382 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:17.334351 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-wp9c9_00897aef-01f7-4dcf-91df-46eca1fb734b/main/0.log" Nov 03 00:48:17.547460 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:17.547423 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-bj2k2_b33c63bc-b6b9-412d-bd69-b0e2c3ebf353/main/0.log" Nov 03 00:48:18.515665 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:18.515637 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-b986v_f674fe20-ea05-4246-a4ff-7cd1da321621/main/0.log" Nov 03 00:48:18.742599 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:18.742569 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-69xh4_359b19c7-a9c6-4476-be2d-9ab428699c1c/main/0.log" Nov 03 00:48:27.844313 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:48:27.844275 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:48:35.370660 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:35.370626 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 03 00:48:35.387061 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:35.387036 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 03 00:48:35.483079 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:35.483054 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67svrn_d9ba2c71-9d0f-41f8-8f4c-b3b510183df3/extract/0.log" Nov 03 00:48:35.498011 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:35.497990 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fj4n7_a5cdd7b8-bc10-4788-a37a-2f93c2fbe0a5/console-operator/2.log" Nov 03 00:48:39.651602 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:39.651571 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wbxxp_417d26e1-2a9d-41e3-8940-69626662a2f3/global-pull-secret-syncer/0.log" Nov 03 00:48:40.253223 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:40.253194 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-25msk_ecdc26b5-be99-4928-8fff-203cc2c19ad6/connection-checker/0.log" Nov 03 00:48:40.713270 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:40.713242 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-276wh_12ce41df-43ef-4b00-be8e-5d3fad591eea/connection-checker/0.log" Nov 03 00:48:41.158025 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:41.157999 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-5wnxz_a8d56c63-207c-42b3-8946-c993d0fde66b/connection-checker/0.log" Nov 03 00:48:42.597365 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:42.597338 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-r5lnw_0ea8d3ef-20e9-4ecb-9e50-2df15d7da726/konnectivity-agent/0.log" Nov 03 00:48:42.988243 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:42.988170 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-176.us-west-1.compute.internal_38d433b21a6573d8bddad58877b72617/haproxy/0.log" Nov 03 00:48:43.512262 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:43.512233 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7fcfb78cb5-m2vnt_b2101b43-6e2c-42d8-8d60-d44faa6433f1/manager/0.log" Nov 03 00:48:43.704215 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:43.704183 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qprz9_b7266a2f-427b-47f5-9528-e8c111ae8d04/manager/0.log" Nov 03 00:48:43.966251 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:43.966210 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-knpxk_f601f493-313e-40b5-9e89-0c492a35255a/manager/0.log" Nov 03 00:48:45.031705 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:45.031675 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-cnzpg_b86548a3-cf54-41a9-8c65-f543ae9daa83/server/0.log" Nov 03 00:48:46.795039 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:46.795003 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-cb6tl_54f62e96-134f-44b3-839e-a02edd85ab08/proxy/0.log" Nov 03 00:48:48.638407 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:48.638372 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-57f5d89f7c-zkm62_cf7f72a3-5ca3-4f5e-a8e2-c88f1485806a/authorino/0.log" Nov 03 00:48:49.943403 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:49.943372 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-ptd66_de8616c3-ba9c-4b99-813e-ef4c190fc4c7/manager/0.log" Nov 03 00:48:50.167110 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:50.167075 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7fcfb78cb5-m2vnt_b2101b43-6e2c-42d8-8d60-d44faa6433f1/manager/0.log" Nov 03 00:48:50.358674 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:50.358644 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-cp9j8_e5198e9e-a261-4491-93da-4fad7b359312/manager/0.log" Nov 03 00:48:50.359195 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:50.359177 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qprz9_b7266a2f-427b-47f5-9528-e8c111ae8d04/manager/0.log" Nov 03 00:48:50.424540 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:50.424507 5727 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t6hpv/perf-node-gather-daemonset-896tz"] Nov 03 00:48:50.425155 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:50.425116 5727 cpu_manager.go:414] "containerMap: removing container" podUID="ec588183-be39-4d98-b4bc-85d74ae54e26" containerName="maas-api" Nov 03 00:48:50.425319 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:50.425300 5727 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec588183-be39-4d98-b4bc-85d74ae54e26" containerName="maas-api" Nov 03 00:48:50.425431 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:50.425420 5727 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec588183-be39-4d98-b4bc-85d74ae54e26" containerName="maas-api" Nov 03 00:48:50.428755 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:50.428739 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-896tz" Nov 03 00:48:50.482079 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:50.482046 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t6hpv\"/\"kube-root-ca.crt\"" Nov 03 00:48:50.517538 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:50.517502 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/84a19382-01c9-44f9-8c78-fba74007f05e-podres\") pod \"perf-node-gather-daemonset-896tz\" (UID: \"84a19382-01c9-44f9-8c78-fba74007f05e\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-896tz" Nov 03 00:48:50.517713 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:50.517549 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmg47\" (UniqueName: \"kubernetes.io/projected/84a19382-01c9-44f9-8c78-fba74007f05e-kube-api-access-lmg47\") pod \"perf-node-gather-daemonset-896tz\" (UID: \"84a19382-01c9-44f9-8c78-fba74007f05e\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-896tz" Nov 03 00:48:50.517713 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:50.517636 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84a19382-01c9-44f9-8c78-fba74007f05e-sys\") pod \"perf-node-gather-daemonset-896tz\" (UID: \"84a19382-01c9-44f9-8c78-fba74007f05e\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-896tz" Nov 03 00:48:50.517713 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:50.517676 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84a19382-01c9-44f9-8c78-fba74007f05e-lib-modules\") pod \"perf-node-gather-daemonset-896tz\" (UID: \"84a19382-01c9-44f9-8c78-fba74007f05e\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-896tz" Nov 03 00:48:50.517896 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:50.517777 5727 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/84a19382-01c9-44f9-8c78-fba74007f05e-proc\") pod \"perf-node-gather-daemonset-896tz\" (UID: \"84a19382-01c9-44f9-8c78-fba74007f05e\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-896tz" Nov 03 00:48:50.531259 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:50.531235 5727 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t6hpv\"/\"default-dockercfg-pwrpp\"" Nov 03 00:48:50.531259 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:50.531239 5727 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t6hpv\"/\"openshift-service-ca.crt\"" Nov 03 00:48:50.533927 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:50.533900 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t6hpv/perf-node-gather-daemonset-896tz"] Nov 03 00:48:50.613866 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:50.613782 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-knpxk_f601f493-313e-40b5-9e89-0c492a35255a/manager/0.log" Nov 03 00:48:50.618510 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:50.618487 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/84a19382-01c9-44f9-8c78-fba74007f05e-podres\") pod \"perf-node-gather-daemonset-896tz\" (UID: \"84a19382-01c9-44f9-8c78-fba74007f05e\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-896tz" Nov 03 00:48:50.618610 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:50.618532 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmg47\" (UniqueName: \"kubernetes.io/projected/84a19382-01c9-44f9-8c78-fba74007f05e-kube-api-access-lmg47\") pod \"perf-node-gather-daemonset-896tz\" (UID: \"84a19382-01c9-44f9-8c78-fba74007f05e\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-896tz" Nov 03 00:48:50.618652 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:50.618607 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/84a19382-01c9-44f9-8c78-fba74007f05e-podres\") pod \"perf-node-gather-daemonset-896tz\" (UID: \"84a19382-01c9-44f9-8c78-fba74007f05e\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-896tz" Nov 03 00:48:50.618652 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:50.618640 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84a19382-01c9-44f9-8c78-fba74007f05e-sys\") pod \"perf-node-gather-daemonset-896tz\" (UID: \"84a19382-01c9-44f9-8c78-fba74007f05e\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-896tz" Nov 03 00:48:50.618721 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:50.618663 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84a19382-01c9-44f9-8c78-fba74007f05e-lib-modules\") pod \"perf-node-gather-daemonset-896tz\" (UID: \"84a19382-01c9-44f9-8c78-fba74007f05e\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-896tz" Nov 03 00:48:50.618721 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:50.618697 5727 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/84a19382-01c9-44f9-8c78-fba74007f05e-proc\") pod \"perf-node-gather-daemonset-896tz\" (UID: \"84a19382-01c9-44f9-8c78-fba74007f05e\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-896tz" Nov 03 00:48:50.618721 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:50.618716 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84a19382-01c9-44f9-8c78-fba74007f05e-sys\") pod \"perf-node-gather-daemonset-896tz\" (UID: \"84a19382-01c9-44f9-8c78-fba74007f05e\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-896tz" Nov 03 00:48:50.618823 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:50.618812 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84a19382-01c9-44f9-8c78-fba74007f05e-lib-modules\") pod \"perf-node-gather-daemonset-896tz\" (UID: \"84a19382-01c9-44f9-8c78-fba74007f05e\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-896tz" Nov 03 00:48:50.618823 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:50.618816 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/84a19382-01c9-44f9-8c78-fba74007f05e-proc\") pod \"perf-node-gather-daemonset-896tz\" (UID: \"84a19382-01c9-44f9-8c78-fba74007f05e\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-896tz" Nov 03 00:48:50.678958 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:50.678930 5727 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmg47\" (UniqueName: \"kubernetes.io/projected/84a19382-01c9-44f9-8c78-fba74007f05e-kube-api-access-lmg47\") pod \"perf-node-gather-daemonset-896tz\" (UID: \"84a19382-01c9-44f9-8c78-fba74007f05e\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-896tz" Nov 03 00:48:50.741590 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:50.741564 5727 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-896tz" Nov 03 00:48:50.773373 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:50.773352 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jr9qg_1394c83b-9059-4e3e-928f-bde172669b09/kuadrant-console-plugin/0.log" Nov 03 00:48:51.117072 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:51.117049 5727 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t6hpv/perf-node-gather-daemonset-896tz"] Nov 03 00:48:51.118746 ip-10-0-135-176 kubenswrapper[5727]: W1103 00:48:51.118709 5727 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod84a19382_01c9_44f9_8c78_fba74007f05e.slice/crio-1ce4d912a75817d72e5e012039138700f2b82a317accc89ff832a57cd5201c9b WatchSource:0}: Error finding container 1ce4d912a75817d72e5e012039138700f2b82a317accc89ff832a57cd5201c9b: Status 404 returned error can't find the container with id 1ce4d912a75817d72e5e012039138700f2b82a317accc89ff832a57cd5201c9b Nov 03 00:48:51.204087 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:51.204050 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-896tz" event={"ID":"84a19382-01c9-44f9-8c78-fba74007f05e","Type":"ContainerStarted","Data":"1ce4d912a75817d72e5e012039138700f2b82a317accc89ff832a57cd5201c9b"} Nov 03 00:48:51.218853 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:51.218829 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-z5dtm_50c690b0-930f-49df-993d-036cd19836a6/manager/0.log" Nov 03 00:48:51.908589 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:51.908561 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-cnzpg_b86548a3-cf54-41a9-8c65-f543ae9daa83/server/0.log" Nov 03 00:48:52.211412 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:52.211320 5727 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-896tz" event={"ID":"84a19382-01c9-44f9-8c78-fba74007f05e","Type":"ContainerStarted","Data":"ed812a29778328f106c0b8096b2a09e76858122200e0deee28a03060a6ac6ae5"} Nov 03 00:48:52.211412 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:52.211390 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-896tz" Nov 03 00:48:52.262058 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:52.262027 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-x6h9v_349fa078-8c82-4c07-a596-900fbb745d7f/manager/1.log" Nov 03 00:48:52.280974 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:52.280945 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-x6h9v_349fa078-8c82-4c07-a596-900fbb745d7f/manager/0.log" Nov 03 00:48:52.335142 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:52.335069 5727 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-896tz" podStartSLOduration=2.335055928 podStartE2EDuration="2.335055928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:52.332163061 +0000 UTC m=+7218.570660714" watchObservedRunningTime="2026-11-03 00:48:52.335055928 +0000 UTC m=+7218.573553589" Nov 03 00:48:53.569344 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:53.569311 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-cb6tl_54f62e96-134f-44b3-839e-a02edd85ab08/proxy/0.log" Nov 03 00:48:56.480351 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:56.480319 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7fcfb78cb5-m2vnt_b2101b43-6e2c-42d8-8d60-d44faa6433f1/manager/0.log" Nov 03 00:48:56.721432 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:56.721405 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qprz9_b7266a2f-427b-47f5-9528-e8c111ae8d04/manager/0.log" Nov 03 00:48:56.982312 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:56.982278 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-knpxk_f601f493-313e-40b5-9e89-0c492a35255a/manager/0.log" Nov 03 00:48:57.836631 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:57.836596 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-qcz4m_53ba063f-0db3-4f82-b280-40718c28b479/cluster-monitoring-operator/0.log" Nov 03 00:48:57.856623 ip-10-0-135-176 kubenswrapper[5727]: E1103 00:48:57.856584 5727 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qlfjm" containerName="extractor" Nov 03 00:48:58.228140 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:58.228058 5727 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-896tz" Nov 03 00:48:58.246105 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:58.246062 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-cnzpg_b86548a3-cf54-41a9-8c65-f543ae9daa83/server/0.log" Nov 03 00:48:59.969932 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:48:59.969895 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-cb6tl_54f62e96-134f-44b3-839e-a02edd85ab08/proxy/0.log" Nov 03 00:49:01.592252 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:49:01.592212 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r5bbz_f0d480ea-7dd1-4e56-93f4-b2007ee9ba54/node-exporter/0.log" Nov 03 00:49:02.005917 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:49:02.005840 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r5bbz_f0d480ea-7dd1-4e56-93f4-b2007ee9ba54/kube-rbac-proxy/0.log" Nov 03 00:49:02.418712 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:49:02.418684 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r5bbz_f0d480ea-7dd1-4e56-93f4-b2007ee9ba54/init-textfile/0.log" Nov 03 00:49:03.500280 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:49:03.500243 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr_856fc499-f20f-41cd-8baf-ed58d33c45c5/istio-proxy/0.log" Nov 03 00:49:06.740767 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:49:06.740729 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-gcxrh_c4c2f4f5-743d-4e9d-ae63-1769c521b4ad/istio-proxy/0.log" Nov 03 00:49:06.747334 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:49:06.747312 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-gcxrh_c4c2f4f5-743d-4e9d-ae63-1769c521b4ad/istio-proxy/1.log" Nov 03 00:49:07.285091 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:49:07.285063 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-kvws4_a5514e1f-1fb4-4158-85c8-f17fe2b94eb8/istio-proxy/0.log" Nov 03 00:49:07.712820 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:49:07.712784 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-vgtv2_3137ed7b-3a02-433a-b95d-7cd75469aac5/istio-proxy/0.log" Nov 03 00:49:07.923069 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:49:07.923041 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-856cdd999f-t96d2_59f85494-1949-4969-8e31-c6a4b155f4ae/istio-proxy/0.log" Nov 03 00:49:08.134935 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:49:08.134905 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-75bc6697f4-778p5_36f33895-59f7-4533-9478-fcd8299a8b4f/payload-pre-processing/0.log" Nov 03 00:49:08.345468 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:49:08.345440 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc7f58bb9-9dh6m_f6ba592a-8ce9-44bd-9ade-8dfd408bad2a/payload-processing/0.log" Nov 03 00:49:08.565213 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:49:08.565186 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-566bb4455-kq2j4_1dccc35b-6206-47f3-b901-bafc5f17b8ad/router/0.log" Nov 03 00:49:14.091914 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:49:14.091877 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-57f5d89f7c-zkm62_cf7f72a3-5ca3-4f5e-a8e2-c88f1485806a/authorino/0.log" Nov 03 00:49:14.205439 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:49:14.205411 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-ptd66_de8616c3-ba9c-4b99-813e-ef4c190fc4c7/manager/0.log" Nov 03 00:49:14.413450 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:49:14.413366 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-cp9j8_e5198e9e-a261-4491-93da-4fad7b359312/manager/0.log" Nov 03 00:49:14.541150 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:49:14.541106 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c4c65fbf-fsgcr_856fc499-f20f-41cd-8baf-ed58d33c45c5/istio-proxy/0.log" Nov 03 00:49:14.623871 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:49:14.623843 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jr9qg_1394c83b-9059-4e3e-928f-bde172669b09/kuadrant-console-plugin/0.log" Nov 03 00:49:14.871462 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:49:14.871429 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-z5dtm_50c690b0-930f-49df-993d-036cd19836a6/manager/0.log" Nov 03 00:49:15.328250 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:49:15.328196 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-x6h9v_349fa078-8c82-4c07-a596-900fbb745d7f/manager/1.log" Nov 03 00:49:15.347063 ip-10-0-135-176 kubenswrapper[5727]: I1103 00:49:15.347035 5727 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-x6h9v_349fa078-8c82-4c07-a596-900fbb745d7f/manager/0.log"